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PREFACE 


The  conduct  of  this  research  and  the  preparation  of  this  report  was  supported  under  a  contract 
designed  to  service  the  Air  Force  Learning  Abilities  Measurement  Program  (LAMP),  a  multi-year 
program  of  basic  research  conducted  at  the  U.S.  Air  Force  Human  Resources  Laboratory  (AFHRL)* . 
and  sponsored  by  the  Air  Force  Office  of  Scientific  Research  (AFOSR).  The  goals  of  this  program  arc 
to  specify  the  basic  parameters  of  learning  ability,  to  develop  techniques  for  the  assessment  of 
individuals’  knowledge  and  skill  levels,  and  to  explore  the  feasibility  of  a  model-based  system  of 
psychological  assessment.  The  present  study  reports  on  applied,  rather  than  basic  research,  and  was 
conducted  to  help  determine  whether  the  new  types  of  computer  measures  coming  out  of  LAMP 
research  show  promise  for  ultimately  improving  the  Ai  Force  personnel  selection  and  classification 
system.  The  author  wishes  to  thank  Scott  Chaiken,  Patrick  Kylloncn,  Dan  Wolf/.,  Bill  Alley.  Valerie 
Shutc,  and  Bob  Young  for  their  review  of  the  draft  manuscript  and  their  helpful  comments.  Thanks 
are  also  given  to  Bill  Tirrc  and  Malcolm  Rcc  for  their  assistance  in  obtaining  data  and  information 
used  in  correcting  validities  and  intcrcorrclations  for  restrictions  in  range  due  to  selection.  Finally, 
special  appreciation  is  expressed  to  Terri  Purdue  who  both  programmed  the  logic  gates  courses  and 
prepared  all  data  for  the  statistical  analyses,  and  to  Janice  Hereford,  who  programmed  data  for  the 
restriction  of  range  corrections. 


*  AFHRL  has  been  redesignated  Human  Resources  Directorate,  Armstrong  Laboratory. 
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COMPARATIVE  VALIDITIES  OF  ASVAB  AND  LAMP  TESTS  FOR  LOGIC 

GATES  LEARNING 


SUMMARY 

Two  short  computer  courses  were  designed  to  teach  basic  airmen  how  to  identify  various  types 
of  logic  gates  and  to  determine  their  output  signals  as  a  function  of  their  input  signals.  In  Study  1,  448 
subjects  received  training  on  simple  gates  with  and  without  negations.  After  training,  they  were 

*  required  to  solve  six  36-item  blocks  of  gate  problems  without  negations,  and  four  36-item  blocks  of  gate 
problems  with  negations.  Instructive  feedback  was  provided  after  each  incorrect  response.  In  Study  2, 
431  subjects  were  trained  on  simple  gates  and  required  to  solve  eight  36-item  blocks  of  simple  gate 
problems  without  negations,  and  one  30-item  block  of  simple  gate  problems  with  negations.  Instructive 
feedback  was  given  after  each  incorrect  response.  These  blocks  were  followed  by  two  60-item  blocks  of 
mixed  simple  gates  (with  and  without  negations),  training  on  complex  gate  circuits,  and,  finally,  four  54- 
item  blocks  of  mixed  simple  gates  and  complex  gate  circuits. 

Subject  responses  were  scored  both  for  accuracy  and  for  latency.  Accuracy  scores  were  used 
as  criteria  indicating  the  acquisition  of  declarative  knowledge  concerning  the  rules  for  solving  logic  gate 
problems.  Latency  scores  at  each  stage  were  used  as  criteria  indicating  the  level  of  procedural  skill 
acquired  up  to  that  stage.  In  each  study,  the  subjects  were  administered  LAMP  tests  designed  to 
measure  working  memory  capacity  and  information  processing  speed.  ASVAB  test  scores  were 
obtained  for  subjects  from  operational  files. 

One  of  the  primary  purposes  of  the  study  was  to  determine  the  relative  power  of  the  the 
ASVAB  tests  and  of  the  LAMP  tests  in  predicting  the  learning  criteria  from  the  logic  gate  courses.  It 
was  also  considered  important  to  determine  how  much  unique  valid  variance  the  LAMP  tests  could 
contribute  to  the  ASVAB  tests  in  predicting  these  criteria.  In  general,  the  LAMP  tests  were  found  to 
.  add  about  20%  unique  valid  variance  to  the  ASVAB  in  predicting  both  accuracy  and  latency  block 

averages  in  the  two  courses.  When  the  intcrcorrelations  and  validities  were  corrected  for  restrictions  of 

*  range  due  to  selection,  the  estimated  contributions  of  LAMP  tests  to  ASVAB  tests  in  accounting  for 
criterion  variance  ranged  from  17%  -  21%  in  the  U.S.Air  Force  (USAF)  applicant  sample,  and  from 
13%  -  20%  in  the  1980  youth  sample.  Furthermore,  against  all  criteria,  LAMP  tests  alone  accounted 
for  more  variance  than  did  ASVAB  alone.  Analyses  were  conducted  in  a  third  study  which  indicated 
that  the  advantage  of  the  LAMP  tests  over  the  ASVAB  tests  was  not  the  result  of  a  concurrency  effect, 
nor  was  it  obtained  at  the  expense  of  excessive  testing  time.  The  results  are  viewed  as  encouraging 
with  respect  to  the  power  of  LAMP  tests  to  predict  meaningful  learning  criteria. 
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INTRODUCTION 


The  Learning  Abilities  Measurement  Program  (LAMP)  is  a  basic  research  program  jointly 
sponsored  by  the  Air  Force  Office  of  Scientific  Research  (AFOSR)  and  the  Air  Force  Human 
Resources  Laboratory  (AFHRL).  It  supports  studies  on  the  identification  and  measurement  of 
abilities  which  can  account  for  individual  differences  in  the  acquisition  of  knowledge  and  skills.  The 
present  framework  for  the  LAMP  program  is  loosely  modeled  after  Anderson’s  (1983)  ACT*  theory, 
and  hypothesizes  that  individual  differences  in  learning  efficiency  arise  from  four  primary  sources: 
a)  the  breadth  and  depth  of  declarative  knowledge  brought  to  the  learning  situation;  b)  the  breadth 
and  depth  of  procedural  skills  brought  to  the  learning  situation;  c)  information  processing  speed 
capabilities;  and  d)  working  memory  capacity  (Kyltonen,  1985b).  Research  during  the  past  several 
years  has  been  focused  on  defining,  measuring  and  validating  abilities  relating  to  processing  speed  and 
working  memory  capacity  (Kyllonen,l985a;  Kyllonen  &  Christal,  1989;  Kyllonen  &  Christal,  1990; 
Kyllonen,  Tit  re,  &  Christal,  1988;  Woltz,  1988). 

Although  LAMP  is  a  basic  research  program,  it  is  nevertheless  goal  oriented.  The  ultimate 
goal  is  to  improve  the  U.S.  Air  Force  (USAF)  personnel  selection  and  classification  system.  In  order 
to  gauge  whether  progress  is  being  made  toward  this  goal,  it  is  desirable  to  periodically  validate  new 
tests  against  meaningful  learning  criteria.  Ultimately,  there  will  be  a  large-scale  validation  against 
school  and  job  performance  criteria.  However,  it  may  be  some  time  before  the  results  of  basic 
research  will  warrant  this  type  of  costly  validation  effort.  In  the  meantime,  in  order  to  generate 
meaningful  learning  criteria  for  test  validation,  the  approach  has  been  to  train  individuals  to  proficiency 
on  operational  tasks  which  arc  brought  into  the  laboratory.  A  new  experimental  training  facility,  the 
Complex  Learning  Assessment  (CLASS)  laboratory,  has  been  established  for  this  purpose.  Individuals 
participating  in  CLASS  experiments  spend  approximately  1  to  1  1/2  days  taking  experimental  LAMP 
tests,  and  then  up  to  30  hours  taking  a  training  course  by  means  of  intelligent  tutors  administered  by 
microcomputers.1  Rich  information  concerning  skill  and  knowledge  acquisition  is  recorded  during 
course  administration,  which,  in  turn,  is  used  to  investigate  learning  behaviors  and  to  validate  scores 
from  the  concurrently  administered  experimental  LAMP  tests.  So  far,  data  have  been  collected  from 
two  such  courses,  one  in  computer  programming  and  one  in  electrical  circuits.  A  third  course,  dealing 
with  fright-engineering  tasks,  is  being  constructed  (Shutc,  in  press-a;  Shutc,  in  press-b;  Shule  <&  Glaser, 
1990,  Shute  &  Pena  ,  1990). 


^here  are  wide  individual  differences  in  both  the  time  required  to  take  the  experimental  tests  and  to 
complete  the  training  courses. 
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Although  data  from  CLASS  experiments  arc  extremely  useful,  developing  and  administering 
intelligent  tutors  is  a  somewhat  slow  and  expensive  process.  It  requires  more  than  a  year  to  develop 
and  administer  a  course.  To  supplement  this  effort,  attempts  have  been  made  recently  to  develop  short 
(1-  to  2-hour)  courses,  on  narrow  subjects,  which  can  be  administered  to  samples  of  airmen  in  Basic 
Military  Training.  The  LAMP  project  has  access  to  samples  of  basic  airmen  for  approximately  3  hours 
of  experimental  testing  during  their  11th  training  day.  This  paper  describes  the  development  of  courses 
on  electronic  logic  gales,  which  were  used  in  three  studies  to  validate  LAMP  tests  and  tests  from  the 
Armed  Services  Vocational  Aptitude  Battery  (ASVAB). 

Logic  gates  were  chosen  for  the  course  subject-matter  for  several  reasons.  First,  they  have 
been  used  by  other  investigators  to  study  skill  acquisition  (Carlson,  Khoo,  Yaurc  &  Schneider,  1990). 
Second,  they  lend  themselves  to  the  development  of  test  items  at  various  difficulty  levels.2  Third,  gate 
problems  were  predicted  to  be  well  suited  for  studies  on  procedural  skill  development.  Finally,  and 
this  is  the  most  important  reason,  previous  research  has  demonstrated  that  knowledge  of  logic  gates  is 
a  distinguishing  factor  between  "skilled  and  less-skilled"  lst-term  airmen  working  in  avionics 
(Gitomcr,1984).  In  light  of  these  observations,  it  was  judged  that  a  short  course  on  logic  gates  would 
fill  the  requirement  for  a  meaningful  learning  task  which  could  be  administered  under  laboratory 
conditions  in  less  than  2  hours. 


GENERAL  APPROACH 

The  present  paper  reports  the  results  of  three  studies.  The  first  half  of  Study  1  involved 
training  on  AND,  OR,  and  XOR  gates,  followed  by  six  blocks  of  test  questions  on  simple  gates  without 
negations.  In  the  second  half  of  Study  1  course  instructions  were  provided  on  the  effects  of  negations, 
and  included  four  blocks  of  questions  on  simple  gates  with  negations.  In  both  sections,  instructive 
feedback  was  provided  after  each  incorrect  response.3 


2Simple  AND  or  OR  gates  without  negations  are  easy.  XOR  gates  are  somewhat  more  difficult.  Gates 
with  negations  are  considerably  more  difficult  to  learn  than  arc  gates  without  negations.  Complex 
circuits  involving  several  gates  can  be  very  difficult  to  solve. 

3  A  distinction  is  made  between  testing  blocks  and  analysis  blocks.  Study  1  included  three  72-itcm 
testing  blocks  of  simple  gates  without  negations  and  two  72-item  testing  blocks  of  simple  gates  with 
negations.  For  analysis  purposes,  these  testing  blocks  were  split  into  six  and  four  36-item  blocks, 
respectively.  Study  2  included  four  72-item  blocks  of  simple  gates  without  negations,  which  were  split 
into  eight  36-item  blocks  for  the  analysis.  In  all  other  instances,  testing  blocks  were  identical  to  analysis 
blocks.  Summary  feedback  was  provided  to  subjects  after  each  testing  block. 
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In  Study  2,  subjects  were  given  a  pre-test  to  determine  their  level  of  "going-in"  knowledge. 
This  was  followed  by:  (a)  instructions  on  simple  AND,  OR,  and  XOR  gates;  (b)  eight  blocks  of  lest 
questions  on  simple  gates  without  negations,  with  instructive  feedback  after  incorrect  responses;  (c) 
instructions  on  negations;  (d)  one  block  of  simple  gates  with  negations,  with  instructive  feedback  after 
incorrect  responses;  (c)  two  blocks  of  mixed  simple  gates,  with  and  without  negations;  (I)  instructions 
on  complex  gate  circuits;  and  (g)  three  blocks  of  test  questions  on  mixed  simple  and  complex  gates, 
some  with  and  some  without  negations. 

In  both  Study  l  and  Study  2,  subjects  were  administered  experimental  tests  developed  in  the 
LAMP  program.  Scores  from  the  ASVAB  were  obtained  for  each  subject  from  operational  files.  A 
series  of  analyses  was  conducted  to  determined  how  well  individual  differences  in  the  acquisition  of 
declarative  knowledge  (reflected  in  accuracy  scores)  could  be  predicted  by  LAMP  tests  alone  and  by 
ASVAB  tests  alone.  Analyses  were  also  conducted  to  sec  how  much  predictive  validity  the  LAMP  tests 
added  to  ASVAB  tests  in  the  prediction  of  gates  course  criteria.  Similar  analyses  were  conducted  to 
determine  the  power  of  the  LAMP  and  ASVAB  tests  to  predict  the  development  of  procedural  skill, 
which  was  reflected  in  the  time  required  to  solve  gate  problems. 

One  of  the  concerns  in  Studies  1  and  2  was  that  the  LAMP  tests  were  administered 
concurrently  (in  the  same  test  session)  with  the  gates  course,  while  the  ASVAB  tests  had  been 
administered  several  months  earlier  in  operational  testing  stations.  Previous  research  has  suggested 
that  concurrent  administration  of  predictor  and  criterion  measures  may  lead  to  a  minor  inflation  in 
estimated  validity  coefficients  (Christal,  1989;  Divgi,  1990).  Study  3  was  conducted  to  evaluate  the 
magnitude  of  this  concurrency  effect,  ft  involved  the  re-administration  of  several  ASVAB  tests  with 
the  Study  2  gates  course.  This  made  possible  a  comparison  of  predictive  (non-concurrent)  and 
concurrent  validity  coefficients  for  the  same  criteria. 

More  detailed  descriptions  of  the  gates  courses  arc  given  in  the  sections  which  follow,  and 
abbreviated  descriptions  of  the  LAMP  and  ASVAB  tests  arc  provided  in  Appendix  A. 


METHOD 


Subjects 


For  Study  1,  the  subjects  were  448  military  recruits  who  were  tested  on  the  6th  day  of  their 
basic  military  training  at  Lackland  Air  Force  Base,  Texas.  Of  these,  94%  were  males  and  6%  females. 
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Studies  2  and  3.  included  431  and  312  subjects,  respectively,  who  were  tested  on  the  11th  day  of  their 
basic  military  training.  In  Study  2,  7.6%  of  the  subjects  were  females,  while  in  Study  3,  28.2%  were 
females.  The  larger  proportion  of  females  in  Study  3  was  fortuitous.  Testing  samples  were  drawn  from 
training  flights  as  they  arrived,  and  such  flights  were  either  all-male  or  all-female.  Not  included  in  the 
above  samples  were  subjects  for  whom  ASVAB  scores  could  not  be  obtained,  those  who  did  not 
complete  the  course  and  experimental  tests  in  the  available  testing  time,  and  up  to  1%  who  were 
eliminated  because  they  were  extreme  outliers  on  the  course  on  one  or  more  of  the  tests. 


Testing  Facility 

The  testing  facility  consisted  of  partitioned  testing  stations  in  a  large  air-conditioned  room. 
Each  station  was  equipped  with  a  Zenith  Z-248  microcomputer  with  a  20  Mb  hard  disk  for  storing 
course  materials,  test  items,  and  subject  responses.  A  standard  keyboard  was  used  for  response  entry, 
and  an  enhanced  graphics  adaptor  (EGA)  and  display  monitor  (640  x  350  color  resolution)  was  used 
for  stimulus  presentation. 

Procedure 

For  each  testing  session,  a  proctor  briefed  subjects  on  the  purposes  of  the  testing,  and  then 
assigned  them  to  testing  stations.  A  computer  program  provided  subjects  with  general  information 
about  the  LAMP  program;  described  use  of  the  keyboard;  told  how  to  ask  questions  of  the  proctor; 
provided  information  about  breaks;  warned  against  talking  with  others  about  the  test  during  breaks; 
and  indicated  what  to  do  when  the  course  and  all  tests  were  completed. 

The  subjects  read  and  responded  to  information  in  accordance  with  the  Privacy  Act.  They 
entered  their  Social  Security  Number,  which  was  required  for  use  in  acquiring  their  ASVAB  scores, 
and  answered  questions  concerning  their  gender,  education,  and  race.  Then  they  responded  to  a  short 
questionnaire  concerning  the  use  of  English  as  a  first  language  at  home  and  in  school. 

In  each  of  the  three  studies,  the  subjects  were  given  a  general  description  of  logic  gates,  and 
answered  12  questions  concerning  whether  they  had  studied  logic  gates  in  school,  had  read  about  them 
in  books,  magazines,  or  newspapers,  knew  how  they  operate,  or  had  used  them  in  building  electronic 
projects.  They  also  answered  questions  concerning  prior  experience  with  microcomputers,  interest  in 
electronics  and  science,  and  whether  they  had  sought  an  assignment  in  a  USAF  electronics  career 
ladder. 
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In  Studies  2  and  3,  the  above  questionnaire  was  administered  immediately  before  or 
immediately  after  a  pre-test  designed  to  measure  subjects’  levels  of  incoming  knowledge  specifically 
relating  to  logic  gates.  These  gate  problems  were  taken  directly  from  the  course  which  was  to  follow, 
except  that  additional  foils  (distractors)  were  generated  to  attract  individuals  who  had  little  or  no 
knowledge  concerning  logic  gates. 

In  all  three  studies,  the  subjects  were  provided  with  a  short  training  course  on  the  shapes  of 
AND,  OR,  and  XOR  gates,  and  were  asked  to  associate  the  names  with  their  respective  shapes.  This 
gate  familiarization  was  followed  with  a  30-item  test  in  which  gates  were  presented  on  the  screen  and 
the  subjects  were  asked  to  indicate  the  associated  name  by  pressing  an  appropriate  key.  After  each 
incorrect  response,  the  subjects  were  informed  of  their  incorrect  response  and  were  presented,  for  10 
seconds,  with  a  screen  which  displayed  all  three  gates  and  their  associated  names.  In  all  three  studies, 
the  first  two  sections  of  the  course  were  identical,  except  for  the  number  of  test  problems  and  the 
method  used  for  timing  instruction  frames.  Each  successive  screen  was  displayed  for  a  specific  period, 
and  then  the  next  screen  automatically  appeared.  The  keyboard  was  ‘locked-out’,  except  during  the 
times  the  subject  was  required  to  enter  responses  to  problems.  In  Study  1,  the  direction  screens 
contained  statements  concerning  time  allowances  (c.g.  "You  will  have  45  seconds  to  study  information 
on  this  screen.").  In  Studies  2  and  3,  these  time  allowance  statements  were  replaced  by  timing  bars  at 
the  top  of  the  screen,  which  became  shorter  with  the  passing  of  each  second.  Otherwise,  the  directions 
for  the  first  two  sections  were  identical  in  all  three  studies,  and  can  be  summarized  as  follows: 

(1)  A  45-second  screen  describing  logic  gates  and  displaying  a  gate  drawing  with  labeled  input 
leads,  gate  body  and  output  lead. 

(2)  A  45-sccond  screen  describing  the  input-output  rules  for  the  AND  gate,  with  two  examples 
provided. 

(3)  A  screen  providing  six  labeled  examples  of  AND  gates,  some  with  two  and  some  with  three 
input  leads.  In  Study  1,  these  examples  were  presented  all  at  once  on  a  45-sccond  screen.  In  Studies  2 
and  3,  these  examples  were  added  to  a  screen  onc-at-a-timc,  after  specified  periods  of  >ime.  In  all 
studies,  the  correct  output  signals  (problem  answers)  were  provided. 

(4)  The  subjects  were  given  a  test  containing  eight  AND  gates,  in  which  they  indicated  whether 
the  output  signal  was  HI  or  LQ  by  pressing  the  ‘H’  or  ‘L’  key.  After  each  incorrect  response,  the 
subject  was  shown  a  screen  for  ten  seconds  which  presented  rules  for  the  particular  gate-type  in 
question. 
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(5)  The  subjects  were  shown  a  scries  of  instruction  screens  for  OR  gates.  This  was  followed 
by  an  8-problcm  test  on  OR  gates,  with  a  10-second  rule-feedback  for  each  item  missed. 

(ft)  The  subjects  were  shown  a  series  of  instruction  screens  for  XOR  gates.  This  was  followed 
by  an  8-problem  test  on  XOR  gales,  with  a  10-second  rule-feedback  for  each  item  missed. 

(7)  The  rules  for  AND,  OR,  and  XOR  gates  were  repeated. 

(8)  In  Study  1,  the  subjects  solved  six  36-itcm  blocks  of  mixed  AND,  OR,  and  XOR  gates, 
without  negations.  After  each  incorrect  response,  the  subject  was  provided  with  a  10-second  instructive 
feedback  frame  which  displayed  the  pictures  and  rules  for  all  three  gate-types,  with  the  missed  gate- 
type  highlighted.  Studies  2  and  3  differed  only  in  the  fact  that  they  included  eight  blocks  (as  opposed  to 
six  in  Study  1),  and  the  feedback  screens  contained  timing  bars.  In  all  studies,  summary  information 
concerning  percent  correct  and  median  response  latency  was  given  at  the  end  of  each  testing  block. 

(9)  Instead  of  complicating  the  study  with  definitions  of  NAND,  NOR  gates  and 
INVERTERS,  the  present  studies  made  use  of  negation  symbols  on  input  and  output  leads.  The 
subjects  were  first  taught  the  impact  of  a  negation  on  the  output  signals  of  AND,  OR,  and  XOR  gates, 
with  six  examples  provided.  Then  the  impact  of  negations  on  input  leads  was  explained,  again  with  six 
examples  given.  Finally,  an  example  was  given  with  negations  on  both  input  and  output  leads.  As  in 
the  case  of  simple  gates  without  negations,  the  study  frames  were  presented  for  specific  lime  periods, 
with  liming  bars  used  in  Studies  2  and  3.  In  Study  1,  the  subjects  were  administered  four  problem 
blocks,  in  which  all  items  contained  negations  on  the  input  leads,  output  leads,  or  both  input  and  output 
leads.  Latency  feedback  was  given  for  correct  responses,  and  instructive  feedback  was  given  for 
incorrect  responses.  Summary  feedback  on  percent  correct  and  median  latency  was  provided  after 
each  testing  block.  In  Studies  2  and  3,  only  one  30-item  test  block  of  gates  with  negations  was  given. 

In  Studies  2  and  3,  the  one  block  of  simple  gates  with  negations  was  followed  by  two  60-item 
blocks  containing  a  mixture  of  simple  gates  with  and  without  negations.  Then  the  subjects  were  given 
instructions  on  how  to  determine  the  final  output  signal  of  complex  circuits  involving  two  input  gates, 
with  or  without  negations,  and  one  output  lead  with  or  without  negations.  Finally  three  54-problem 
blocks  of  were  presented,  each  of  which  contained  a  mixture  of  simple  gates  (with  and  without 
negations),  and  complex  gales  (with  and  without  negations).  Feedback  in  the  form  of 
CORRECT/WRONG  and  response  latency  was  provided  for  each  of  the  items  in  the  mixed  simple 
gates  and  the  mixed  simple/complcx  gates  blocks,  along  with  summary  data  at  the  end  of  each  testing 
block  Although  details  will  not  be  provided,  the  blocks  within  sections  were  balanced  in  terms  of  item 
types. 
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Figure  1  presents  the  abbreviations  for  the  predictor  tests,  and  Figure  2  presents  abbreviations 
for  the  course  criteria.  Both  sets  of  abbreviations  are  used  in  tables  throughout  the  remaining  part  of 
this  paper. 


Figure  1.  Test  Abbreviations,  Titles  and  Study  Usage 


Abbreviation 

Title 

Study  1 

Study2 

Study3 

ASVAB 

GS 

General  Science 

X 

X 

AR 

Arithmetic  Reasoning 

X 

X 

X 

ARR 

Arithmetic  Reasoning,  Retest 

X 

WK 

Word  Knowledge 

X 

X 

PC 

Paragraph  Comprehension 

X 

X 

NO 

Numerical  Operations 

X 

X 

CS 

Coding  Speed 

X 

X 

AS 

Auto  and  Shop  Information 

X 

X 

MK 

Mathematics  Knowledge 

X 

X 

X 

MKR 

Mathematics  Knowledge,  Retest 

X 

MC 

Mechanical  Comprehension 

X 

X 

X 

MCR 

Mechanical  Comprehension,  Retest 

X 

El 

Electrical  Information 

X 

X 

LAMP 

CRA 

Choice  Reaction,  Accuracy 

X 

X 

CRL 

Choice  Reaction,  Latency 

X 

X 

PIA 

Physical  Identity,  Accuracy 

X 

X 

PIL 

Physical  Identity,  Latency 

X 

X 

NIA 

Name  Identity,  Accuracy 

X 

X 

NIL 

Name  Identity,  Latency 

X 

X 

MIA 

Meaning  Identity,  Accuracy 

X 

MIL 

Meaning  Identity,  Latency 

X 

NFA 

Number  Fact  Retrieval,  Accuracy 

X 

NFL 

Number  Fact  Retrieval,  Latency 

X 

ABA 

Grammatical  Reasoning,  Accuracy 

X 

X 

APL 

Grammatical  Reasoning,  Latency 

X 

X 

NRA 

Number  Reduction,  Accuracy 

X 

DSA 

Digit  Span,  Accuracy 

X 

NAA 

ABC  Numerical  Assignment,  Accuracy 

X 

X 

GKA 

General  Knowledge,  Accuracy 

X 

GKL 

General  Knowledge,  Latency 

X 

WAA 

Word  Association,  Accuracy 

X 

FRA 

Figure  Recognition,  Accuracy 

X 

MCA 

Mental  Character  Generation,  Accuracy 

X 

Notes:  See  Appendix  A  for  lest  descriptions. 
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FigurcZCodcs^efinilionSjandjil^^ 


Code 

-  ’  . -  — -p .  -p . 

Description  Studyl 

Study2 

Study3 

1-1 

Simple  Gates,  No  Negations,  Block  1 

36 

36 

36 

1-2 

Simple  Gates,  No  Negations,  Block  2 

36 

36 

36 

1-3 

Simple  Gates,  No  Negations,  Block  3 

36 

36 

36 

1-4 

Simple  Gates,  No  Negations,  Block  4 

36 

36 

36 

1-5 

Simple  Gates,  No  Negations,  Block  5 

36 

36 

36 

1-6 

Simple  Gates,  No  Negations,  Block  6 

36 

36 

36 

Avl 

Simple  Gales,  No  Negations,  Average,  Blks  1-6 

216 

1-7 

Simple  Gates,  No  Negations,  Block  7 

36 

36 

1-8 

Simple  Gates,  No  Negations,  Block  8 

36 

36 

Avl 

Simple  Gates,  No  Negations,  Average,  Blks  1-8 

288 

288 

2-1 

Simple  Gales,  With  Negations,  Block  1 

36 

30 

30 

2-2 

Simple  Gates,  With  Negations,  Block  2 

36 

2-3 

Simple  Gates,  With  Negations,  Block  3 

36 

2-4 

Simple  Gates,  With  Negations,  Block  4 

Av2 

Simple  Gates,  With  Negations,  Blks  1-4 

144 

3-1 

Mixed  Simple  Gates,  With  &  Without  Negations,  Block  1 

60 

60 

3-2 

Mixed  S'mple  Gates,  With  &  Without  Negations,  Block  2 

60 

60 

Av3 

Mixed  Simple  Gales,  With  &  Without  Negations,  Blks  1-2 

120 

120 

4-1 

Mixed  Simple  &  Complex  Gates,  With  &  Without  Neg.  Blk  1 

54 

54 

4-2 

Mixed  Simple  &  Complex  Gates,  With  &  Without  Neg.  Blk  2 

54 

54 

4-3 

Mixed  Simple  &  Complex  Gates,  With  &  Without  Neg.  Blk  3 

54 

54 

Av4 

Mixed  Simple  &  Complex  Gates,  With  &  Without  Neg.  Avg.  1-3 

162 

162 

avc; 

Average  Across  Ail  Course  Blocks 

360 

600 

600 

Notes:  Codes  are  used  in  tables  throughout  the  paper  to  identify  blocks  of  accuracy  and  latency  scores. 
The  Avl  values  arc  averages  across  six  blocks  in  Study  1,  and  across  eight  blocks  in  Study  2.  Table 
values  indicate  the  number  of  items  in  the  various  blocks  and  block  averages. 
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STUDY  1 


Learning  Performance 

Study  1  involved  administration  of  six  36-probicm  blocks  of  simple  gates  without  negations  and 
four  36-item  blocks  of  simple  gates  with  negations.  Table  1  presents  the  means  and  standard 
deviations  for  the  accuracy  scores  and  for  the  median  latency  scores  for  these  various  blocks. 


Table  1.  Means  and  Standard  Deviations  for  Study  1  Course  Blocks  and  Averages 


BLK  CODE 

1-1 

1-2 

1-3 

1-4 

1-5 

1-6 

Avl 

2-1 

2-2 

warn 

Av2 

AV(5 

ACCURACY 

Mean 

79.2 

87.8 

91.7 

93.2 

94.8 

95.2 

90.3 

83.8 

87.0 

89.2 

89.6 

87.4 

87.') 

S.D. 

11.8 

10.2 

8.1 

7.4 

6.9 

6.4 

6.9 

15.4 

13.8 

12.8 

12.6 

12.6 

7.9 

LATENCY 


Mean 

1.87 

1.58 

1.43 

1.27 

1.21 

1.17 

1.37 

3.39 

2.98 

2.74 

2.74 

2.95 

1.80 

S.D. 

.53 

.43 

.39 

.29 

.29 

.27 

.30 

.87 

.76 

.71 

.73 

.70 

.40 

Notes:  Block  codes  arc  defined  in  Figure  2. 


It  is  clear  from  data  in  Table  1  that  both  declarative  and  procedural  learning  took  place  across 
the  entire  course.  In  the  case  of  simple  gates  without  negations,  the  mean  accuracy  rose  from  79.2%  in 
Block  1  to  95.2%  in  block  6,  while  the  standard  deviations  dropped  from  11.8%  to  6.4%  across  these 
same  blocks.  Since  the  "going-in"  knowledge  about  logic  gates  for  the  basic  airmen  is  near  zero  (see 
Study  2),  the  mean  accuracy  level  of  79.2%  for  problems  in  Block  1  must  be  attributed  to  learning 
which  took  place  in  the  course  prior  to  Block  1,  and  to  the  instructional  feedback  given  for  problems 
missed  within  Block  1.  The  remaining  learning  (reflected  in  the  increase  in  means  from  79.2%  to 
95.2%)  can  be  attributed  to  learning  from  the  instructive  feedback  given  for  problems  missed  after 
Block  1. 
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The  group  data  also  reveal  a  systematic  increase  in  the  development  of  procedural  skill.  The 
mean  of  the  median  response  latencies  dropped  from  1.87  seconds  in  Block  1  to  1.17  seconds  in  Block 
6,  and  the  standard  deviations  for  latencies  dropped  from  .53  to  .27  seconds  across  these  same  Blocks. 

In  the  case  of  gates  with  negations,  the  learning  rales  were  considerably  lower,  rising  from 
83.8%  accuracy  in  Block  1  to  89.6%  accuracy  in  Block  4,  with  the  associated  standard  deviations 
dropping  from  15.4%  to  12.6%.  There  was  some  indication  of  the  development  of  procedural  skill 
during  the  first  three  blocks,  but  reductions  in  latencies  leveled  off  after  Block  3. 

Table  2  reports  the  intcrcorrelatons  of  accuracy  scores  for  all  problem  blocks.  The 
intercorrelation  matrix  for  the  six  blocks  of  gates  without  negations  (labeled  l-l  to  1-6)  was  in  the  form 
of  a  pseudo  simplex,  the  highest  values  being  between  adjacent  blocks  and  the  lowest  values  being 
between  the  blocks  furthermost  removed  from  each  other..  The  same  is  true  for  the  four  blocks  of  gate 
problems  with  negations.  The  magnitudes  of  the  correlations  between  adjacent  blocks  provide  a  rough 
indication  of  the  stability  of  individual  differences  in  performance  across  those  blocks.  The  lower 
intcrcorrelations  between  the  blocks  of  gates  without  negations  can  be  partly  attributed  to  a  restriction 
in  range  due  to  a  ceiling  effect.  In  Blocks  1-6,  31.7%  of  the  subjects  made  perfect  scores,  while  another 
29.9  %  missed  only  one  item. 


Table  2.  Intercorrelations  of  Accuracy  Scores  from  Study  1  Course  Blocks 


VAR 

1-1 

1-2 

1-3 

1-4 

1-5 

1-6 

2-1 

2-2 

2-3 

1-1 

l.(K) 

1-2 

.68 

1.00 

1-3 

.59 

.72 

1.00 

1-4 

.48 

.61 

.69 

1.00 

1-5 

.43 

.56 

.60 

.73 

1.00 

1-6 

.39 

.49 

56 

.68 

.76 

1.00 

2-1 

.39 

.42 

.42 

.44 

.50 

.49 

1.00 

2-2 

.42 

.45 

.50 

.54 

.61 

.58 

.83 

1.00 

2-3 

.42 

.48 

.49 

.59 

.63 

.59 

.76 

.85 

1.00 

2-4 

.44 

.53 

.48 

.58 

.64 

.60 

.74 

.81 

.86 

Notes.  Block  variable  codes  are  defined  in  Figuic  2. 
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Table  3  presents  the  intercorrelations  among  the  latency  scores  for  the  various  learning  blocks. 
Again,  the  values  in  this  matrix  are  in  the  form  a  pseudo  simplex,  with  the  latter  adjacent  blocks  of 
gates  without  negations  correlating  in  the  upper  .80s. 


Table  3,  Intercorrelations  of  Latency  Scores  from  Study  1  Course  Blocks 


VAR 

1-1 

1-2 

1-3 

1-4 

1-5 

1-6 

2-1 

2-2 

2-3 

1-1 

1.00 

1-2 

.74 

1.00 

1-3 

.56 

.75 

1.00 

1-4 

.54 

.73 

.88 

1.00 

1-5 

.48 

.67 

.81 

.85 

1.00 

1-6 

.47 

.62 

.77 

.82 

.88 

1.00 

2-1 

.55 

.56 

.51 

.50 

.48 

.49 

1.00 

2-2 

.51 

.54 

.50 

.52 

.48 

.51 

.86 

1.00 

2-3 

.43 

.46 

.46 

.48 

.44 

.46 

.73 

.80 

1.00 

2-4 

.40 

.42 

.36 

.38 

.36 

.41 

m 

.75 

.80 

Notes.  Block  variable  codes  are  defined  in  Figure  2. 


Facto,  Analysis 

The  names  and  descriptions  of  the  ability  tests  administered  to  subjects  in  Study  1  arc 
presented  in  Appendix  A.  In  the  case  of  ASVAB  tests,  all  scores  were  measures  of  accuracy,  although 
the  Numerical  Operations  and  Coding  Speed  tests  were  administered  under  speeded  conditions,  and 
the  resulting  scores  were  based  on  the  number  of  correct  answers  recorded  during  the  allotted  time. 
Nine  tests  were  selected  from  the  LAMP  program  for  inclusion  in  Study  1,  all  of  which  were  designed 
to  be  measures  of  information  processing  speed  or  working  memory  capacity.  Accuracy  scores  were 
obtained  from  all  of  the  LAMP  tests,  but  latency  scores  were  computed  only  for  those  processing  speed 
tasks  that  yielded  high  accuracy  scores. 

From  the  listing  in  Figure  1,  it  can  be  seen  that  a  total  of  25  test  scores  were  available  for  the 
validation  analyses  (10  accuracy  scores  from  the  ASVAB,  9  accuracy  scores  from  LAMP  tests,  and  6 
latency  scores  from  LAMP  tests).  There  was  interest  in  determining  how  the  common  factor  structure 
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of  the  ASVAB  and  LAMP  tests  correlated  with  the  course  criteria.  Therefore  six  principal  axis  factors 
were  extracted  from  the  25-variable  intercorrelation  matrix  and  rotated  orthogonally  using  the  Varimax 
method.  The  results  of  this  factor  analysis  arc  presented  in  Appendix  B.  Six  factors  were  identified, 
which  were  named  (1)  Processing  Speed;  (2)  Working  Mcmoiy;  (3)  Processing  Accuracy;  (4)  Verbal; 
(5)  Technical  Knowledge;  and  ((>)  Numerical  Facility/Pcrccptual  Speed.  Scores  on  these  factors  were 
correlated  with  accuracy  and  latency  criteria  from  the  logic  gates  course. 


CY  mlation  of  Factor  Scores  with  Learning  Criteria 


Accuracy  Scores 

Factor  scores  were  computed  for  all  subjects  and  correlated  with  accuracy  and  latency  scores 
from  the  various  problem  blocks  in  the  gates  course.  The  results  of  these  analyses  are  presented  in 
Tables  4  and  5.  Data  in  Table  4  clearly  indicate  that  individual  differences  in  the  acquisition  of 
declarative  knowledge  were  highly  associated  with  individual  differences  in  working  memory  capacity. 
This  association  is  not  surprising,  since  the  primary  requirement  of  problem  solving  was  to  hold  a  set  of 
complex  rules  in  memory  while  processing  information  concerning  the  inpul-output  signals  of  gate 
types.  Errors  could  have  beer,  made  either  through  memory  failures  concerning  rules,  or  through  the 
inaccurate  execution  of  processing  steps  required  in  applying  the  rules.  The  working  memory  factor 
(WKMEM)  scores  correlated  in  the  high  30’s  or  low  40’s  with  accuracy  scores  from  the  individual 
problem  blocks,  including  both  those  without  negations  and  those  with  negations.  The  reduction  in  the 
magnitude  of  relationships  with  the  last  few  blocks  of  gate  problems  without  negations  could  have  been 
due  to  the  ceiling  effects  mentioned  previously.  The  Working  Memory  factor  scores  correlated  .50  with 
an  average  accuracy  score  across  all  blocks  involving  gates  without  negations,  .45  with  an  average 
across  blocks  involving  gates  with  negations,  and  .53  with  an  average  across  all  learning  blocks. 


The  Processing  Accuracy  (PROCACC)  factor  scoies  had  the  second  highest  relationships  with 
the  gate-problem  accuracy  stores,  the  correlation  coefficients  generally  running  in  the  mid-  to  upper- 
.20\  for  individual  learning  blocks,  and  low  ,30’s  for  block  averages.  The  Verbal  and  Technical 
Knowledge  factors  had  low  but  significant  relationships  with  accuracy  scores  on  blocks  involving 
negations. 
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Table  4.  Correlations  of  Factor  Scores  with  Study  1  Course  Accuracy  Criteria 
BLOCK  PROCSPD  WKMEM  PROACC  VERBAL  TECHKNOW  NUM/PS 


1-1 

.07 

.42 

.24 

.12 

.04 

.13 

1-2 

.10 

.43 

.27 

.09 

.07 

.09 

1-3 

.14 

.43 

.25 

.07 

.05 

.03 

1-4 

.14 

.40 

.27 

.02 

.09 

.05 

1-5 

.10 

.39 

.32 

-.00 

.04 

.07 

1-6 

.07 

.35 

.30 

.04 

.06 

.07 

Avl 

.12 

.50 

.33 

.08 

.07 

.10 

2-1 

.08 

.43 

.25 

.22 

.20 

.01 

2-2 

.05 

.42 

.28 

.17 

.17 

.00 

2-3 

.08 

.42 

.26 

.16 

.17 

-.02 

2-4 

.06 

.41 

.32 

.12 

.14 

.02 

Av2 

.07 

.45 

.30 

.18 

.18 

.00 

AVG 

.11 

.53 

.35 

.15 

.16 

.06 

Notes.  Block  codes  are  defined  in  Figure  2.  PROCSPD  =  Processing  Speed;  WKMEM  =  Working 
Memory;  PROACC  =  Processing  Accuracy;  TECHKNOW  =  Technical  Knowledge;  NUM/PS  - 
Numerical  Facility-Perceptual  Speed. 

Latency  Scores 

Data  in  Table  5  show  the  relationships  of  factor  scores  with  median  latency  scores  from  the 
logic  gates  course.  The  highest  validities  were  obtained  from  the  Processing  Speed  factor.  Significant 
relationships  were  also  obtained  between  the  Working  Memory  factor  and  latency  criteria  for  the  last 
four  blocks  of  simple  gates  without  negations.  This  correlation  suggests  that  individuals  with  good 
working  memory  capacity  had  sufficient  strength  of  declarative  knowledge  by  this  stage  to  facilitate  the 
development  of  procedural  skill. 
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Tabic  5.  Correlations  of  Factor  Scores  with  Study  1  Course  Latency  Criteria 


BLOCK 

PROCSPD 

WKMEM 

PROACC 

VERBAL  TECHKNOW  NUM/PS 

1-1 

.24 

.06 

.18 

.01 

.00 

-.13 

1-2 

.23 

-.04 

.05 

-.08 

.00 

-.13 

1-3 

.25 

-.16 

.03 

-.03 

-.03 

-.13 

1-4 

.33 

-.15 

.05 

-.04 

-.01 

-.10 

1-5 

.31 

-.19 

.00 

-.04 

-.04 

-.12 

1-6 

.35 

-.18 

.04 

-.02 

-.02 

-.13 

Avl 

.34 

-.12 

.06 

-.04 

-.01 

-.14 

2-1 

.29 

.07 

.17 

.00 

-.01 

-.17 

2-2 

.32 

.01 

.14 

.01 

.04 

-.18 

2-3 

.32 

.02 

.05 

-.02 

.02 

-.18 

2-4 

.34 

.05 

.10 

.06 

.03 

-.20 

Av2 

.35 

.05 

.13 

.02 

.02 

-.19 

AVG 

.35 

-.04 

.12 

-.02 

.01 

-.18 

Notes.  Block  codes  are  defined  in  Figure  2. 
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Significant  relationships  were  also  found  between  the  Numerical  Facility/Pcrccptual  Speed 
factor  and  gatcs-course  latency  criteria.  However,  in  a  subsequent  analysis,  it  was  revealed  that  all  of 
these  relationships  were  accounted  for  by  the  LAMP  Number  Fact  Retrieval  test.  Neither  the  ASVAB 
Numerical  Operations  test  nor  the  Coding  Speed  test  contributed  to  the  prediction  of  latency  criteria. 


Comparative  Validities  of  ASVAB  and  LAMP  Tests  for  Gates-Course  Learning  Criteria. 

A  primary  goal  of  the  present  set  of  studies  was  to  determine  whether  LAMP  tests  would  do 
as  well  or  better  than  conventional  papcr-and-pcncil  tests  in  predicting  meaningful  learning  criteria.  Of 
course,  the  present  studies  were  limited  to  learning  in  a  single  content  arca-that  of  electronic  logic 
gates.  Moreover,  time  permitted  only  a  small  sample  of  LAMP  tests  to  be  administered  and  validated. 
Nevertheless,  results  from  the  present  research  will  add  to  an  accumulation  of  evidence  to  be  used  to 
evaluate  LAMP  tests. 

In  the  logic  gates  task,  two  types  of  criterion  data  were  considered.  The  first  plausibly 
reflected  individual  differences  in  the  acquisition  of  declarative  knowledge  and  was  indexed  by  the 
accuracy  of  specifying  the  outputs  of  logic  gates.  The  second  criterion  reflected  the  acquisition  of 
procedural  skill,  and  was  indexed  by  the  time  required  for  the  subjects  to  correctly  analyze  and  solve 
the  logic  gates  problems.  Each  of  these  aspects  of  the  criterion  was  treated  separately  in  the  following 
analyses. 


Acquisition  of  Declarative  Knowledge 

The  percent  correct  score  for  any  one  block  of  gate  problems  indicated  the  level  of  declarative 
knowledge  for  a  subject  at  that  stage  of  learning.  Thus,  it  was  desirable  to  relate  ASVAB  and  LAMP 
tests  to  accuracy  scores  separately  for  each  of  the  six  blocks  of  gates  without  negations  and  for  each  of 
the  four  blocks  with  negations.  There  were  two  problems  with  this  approach.  First,  as  more  and  more 
individuals  reach  a  level  of  perfect  accuracy,  the  obtained  validities  suffered  from  a  restriction  in  range 
due  to  ceiling  effects.  Second,  individual  block  scores  were  based  on  a  small  number  of  problems,  and 
were  therefore  somewhat  unreliable.  For  this  reason,  the  average  latency  across  each  series  of  blocks 
was  also  included  as  a  criterion. 
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Two  approaches  were  taken  to  test  validation.  Both  involved  computing  differences  in  full- 

and  restricted-model  R2s.  The  initial  step  in  the  Approach  1  analysis  was  to  include  alt  ASVAB  and 

LAMP  tests  in  a  single  multiple  regression  equation  to  predict  each  criterion.  The  squares  of  these 

multiple  correlations  (R2s)  refected  the  amount  of  criterion  variance  accounted  for  by  the  combined 

lest  battery.  Then  the  LAMP  tests  were  removed  from  the  equations  to  determine  the  losses  in 

criterion  variance  accounted  for.  In  each  instance,  the  loss  in  R2  indicated  the  unique  contribution  of 

the  LAMP  tests  to  the  ASVAB  tests  in  accounting  for  criterion  variance.  Next,  the  ASVAB  tests  were 

removed  from  the  full  equation  to  determine  their  unique  contribution  to  LAMP  tests  in  accounting  for 

2 

criterion  variance.  In  both  instances,  the  significance  levels  of  R  losses  were  computed.  The  results 
Irom  these  analyses  are  reported  in  Table  6. 


Table  ft.  Unique  Contributions  of  ASVAB  and  LAMP  Tests  to  the  Prediction  of  Study  1  Accuracy 
Criteria,  Estimated  by  Approach  1 _ 

ASVAB  ADDS  TO  LAMP  LAMP  ADDS  TO  ASVAB  FULL-MODEL 
BLK  R2  F  SIGN.  R2  F  SIGN.  R  R2 


1-1 

.024 

1.397 

.179 

.131 

5.103 

.000 

.528 

.279 

1-2 

.017 

.997 

.445 

.153 

6.037 

.000 

.535 

.287 

1-3 

.010 

.600 

.814 

.180 

7.117 

.000 

.539 

.290 

1-4 

.016 

.904 

.530 

.168 

6.507 

.000 

.525 

.275 

1-5 

.022 

1.371 

.191 

.210 

8.667 

.000 

.565 

.319 

1-f) 

.006 

.345 

.968 

.186 

7.197 

.000 

.521 

.272 

AvI 

.017 

1.153 

.321 

.222 

10.319 

.(XX) 

.628 

.394 

2-1 

.065 

4.269 

.000 

.133 

5.809 

.000 

.596 

.356 

2-2 

.036 

2.2(4 

.014 

.155 

6.421 

.000 

.567 

.321 

2-3 

.034 

2.184 

.018 

.174 

7.370 

.000 

.579 

.336 

2-4 

.025 

1.562 

.115 

.192 

8.071 

.000 

.574 

.329 

Av2 

.044 

3.008 

.001 

.182 

8.267 

.000 

.617 

.381 

AVG 

.035 

2.763 

.003 

.236 

12.412 

.000 

.(>82 

.465 

Notes:  Block  codes  are  defined  in  Figure  2.  See  text  for  explanation  of  Approach  1. 
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The  combination  of  LAMP  and  ASVAB  tests  did  a  reasonably  good  job  of  predicting 
individual  differences  in  learning  efficiency,  the  obtained  multiple  correlations  ranging  from  the  low  to 
the  high  .50’s  for  the  various  learning  blocks.  The  multiple  correlations  for  an  average  across  blocks 
without  negations  was  .628,  for  blocks  with  negations  was  .617,  and  for  all  blocks  combined  was  .682 
(46.5  percent  of  the  criterion  variance  being  accounted  for).  Tests  from  the  ASVAB  did  not  add 
significantly  to  LAMP  tests  in  predicting  accuracy  scores  on  logic  gates  without  negations,  and  added 
very  little  to  the  LAMP  tests  in  predicting  learning  on  gates  with  negations.  On  the  other  hand,  LAMP 
tests  uniquely  accounted  for  an  additional  22.2%  of  the  criterion  variance  for  gates  without  negations, 
18.2%  for  gates  with  negations,  and  23.6%  for  an  average  across  all  the  problem  blocks. 

2 

The  significance  levels  associated  with  losses  in  R  s  reported  in  Table  6  should  be  trustworthy, 
since  appropriate  degrees  of  freedom  were  used  in  the  computations  of  F-ratios.  However,  the 
estimated  unique  contributions,  which  were  determined  by  subtracting  restricted-model  R  s  from  full- 
model  R  s,  were  likely  to  be  somewhat  inaccurate  because  of  the  varying  number  of  independent 
variables  in  the  equations.  Other  factors  held  constant,  the  larger  the  number  of  independent  variables 
in  an  equation,  the  more  likely  the  computed  RZ  will  be  an  overestimate  of  the  population  parameter. 
Since  the  ASVAB  equations  contained  only  ten  independent  variables,  while  the  LAMP  equations 
contained  fifteen,  there  was  a  greater  opportunity  for  capitalization  on  char.ce  relationships  in  the 
LAMP  equations.  There  was  even  a  greater  opportunity  for  capitalization  on  chance  relationships  in 
the  full-model,  which  contained  25  independent  variables.  To  ameliorate  these  circumstances, 
Approach  2  was  taken,  in  which  unique  contributions  were  estimated  using  differences  in  R2s  which 
had  been  adjusted  for  the  number  of  independent  variables  to  more  closely  approximate  the  population 
parameters.4  The  results  of  the  Approach  2  analyses  are  reported  in  Table  7. 

2 

Table  7  reports  the  Rs  and  adjusted  R  s  for  ASVAB  tests  alone,  for  LAMP  tests  alone,  and 
for  a  combination  of  LAMP  and  ASVAB  tests  for  the  various  gates-coursc  accuracy  criteria.  The  last 
column  indicates  the  unique  contribution  of  LAMP  tests  to  the  ASVAB  in  accounting  for  criterion 
variance.5  These  values  arc  based  upon  differences  in  the  full-  and  restricted-model  adjusted  R  s,  and 
therefore  should  be  closer  to  the  population  parameters.  They  indicate  that  the  LAMP  tests  added 
20.5%  unique  variance  to  ASVAB  in  accounting  for  average  accuracy  scores  across  gale  problems 


4A  sample  R2  usually  docs  not  fit  the  population  as  well  as  it  fits  the  computing  sample.  The  following 
equation  was  applied  to  adjust  the  R^  to  more  closely  reflect  the  goodness  of  fit  of  the  model  in  the 
population:  R2a  =  R2  -  (p(l-R2))/(N-p-1),  where  p  is  the  number  of  independent  variables  in  the 
equation. 

5In  this,  as  well  as  in  succeeding  tables,  LAMP/ASVAB  and  ASVAB/LAMP  represent  the 
contributions  of  LAMP  to  ASVAB  and  ASVAB  to  LAMP,  respectively. 
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without  negations,  16.3%  in  accounting  for  average  accuracy  scores  across  gate  problems  with 
negations,  and  22.2%  in  accounting  for  average  accuracy  scores  across  all  ten  problem  blocks. 

While  the  LAMP  tests  contributed  substantially  to  ASVAB  tests  in  accounting  for  the 
individual  differences  in  declarative  learning  as  reflected  in  the  accuracy  scores,  (he  data  indicate  that 
ASVAB  tests  added  nothing  to  the  LAMP  tests  in  predicting  performance  on  the  test  blocks  without 
negations,  and  very  little  to  the  LAMP  tests  in  predicting  performance  on  the  test  blocks  with 
negations. 


Tabic  7.  Unique  Contributions  of  ASVAB  and  LAMP  tests  to  the  Prediction  of  Study  1  Accuracy 
Criteria,  Estimated  by  Approach  2 


BLOCK 

FULL  MODEL 

R  ADJ  R2 

LAMP 

R  ADJ  R2 

ASVAB 

R  ADJ  R2 

LAMP/ 

ASVAB 

ASVAB/ 

LAMP 

1-1 

.528 

.236 

.504 

.229 

.385 

.129 

.107 

.007 

1-2 

.535 

.244 

.519 

.244 

.366 

.114 

.130 

.000 

1-3 

.539 

.248 

.529 

.255 

.332 

.090 

.158 

-.007 

1-4 

.525 

.232 

.509 

.234 

.328 

.087 

,175 

-.002 

1-5 

.565 

.279 

.545 

.272 

.330 

.089 

.190 

.007 

1-6 

.521 

.229 

.516 

.240 

.293 

.065 

.164 

-.011 

Avl 

.628 

.358 

.614 

.356 

.415 

.153 

.205 

.002 

2-1 

.5% 

.317 

.539 

.266 

.412 

.205 

.112 

.051 

2-2 

.567 

.281 

.534 

.260 

.408 

.149 

.132 

.021 

2-3 

.579 

.296 

.549 

.277 

.402 

.143 

.153 

.019 

2-4 

.574 

.290 

.552 

.280 

.370 

.117 

.173 

.010 

Av2 

.617 

.344 

.580 

.314 

.446 

.181 

.163 

.030 

avc; 

.(>81 

.433 

.656 

.410 

.478 

.211 

.222 

.023 

Notes.  In  those  instances  in  which  the  adjusted  R2  for  the  LAMP  tests  is  higher  than  that  for  the 
FULL  MODEL,  the  estimated  contribution  for  the  ASVAB  tests  is  negative.  This  is  an  artifact  of  the 
estimation  procedure.  Block  codes  arc  defined  in  Figure  2.  Sec  text  for  explanation  of  Approach  2. 


Acquisition  of  Procedural  Skill 

In  most  instructional  settings,  the  development  of  procedural  skill  tends  to  follow  the 
acquisition  of  declarative  knowledge.  In  the  present  study,  individuals  first  teamed  the  rules  for 
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analyzing  the  input  signals  for  logic  gates  in  order  to  correctly  specify  their  output  signals.  With 
practice  in  applying  these  rules,  they  were  able  to  increase  the  speed  of  this  analysis  and  decision 
activity.  Thus,  the  development  of  procedural  skill  was  reflected  in  the  latency  scores.  Table  8  reports 
the  results  of  an  analysis  to  determine  the  unique  contributions  of  LAMP  and  ASVAB  tests  in 
accounting  for  variance  in  these  scores.  It  is  similar  to  the  Approach  1  data  presented  in  Table  6  for 
accuracy  scores.  The  multiple  correlations  are  for  the  full  model,  using  a  combination  of  all  ASVAB 
and  LAMP  tests,  while  the  other  columns  present  the  unique  contributions  of  ASVAB  to  I  AMP  tests 
and  LAMP  tests  to  ASVAB  in  accounting  for  the  criterion  variance.  The  F-ratios  and  significance 
levels  are  reasonable  population  estimates,  since  they  were  computed  using  appropriate  degrees-of- 
frecdom  values. 


Table  8.  Unique  Contributions  of  ASVAB  and  LAMP  Tests  to  the  Prediction  of  Study  1  Latency 
Criteria,  Estimated  by  Approach  1 

ASVAB  ADDS  TO  LAMP  LAMP  ADDS  TO  ASVAB  FULL-MODEL 
BLK  R2  F  SIGN.  R2  F  SIGN.  R  R2 


1-1 

.021 

1.13 

.339 

.167 

5.847 

.000 

.443 

.197 

1-2 

.009 

.452 

.920 

.108 

3.542 

.000 

.272 

.138 

1-3 

.011 

.539 

.863 

.104 

3.413 

.000 

.379 

.144 

1-4 

.008 

.430 

.932 

.154 

5.298 

.000 

.428 

.184 

1-5 

.014 

.723 

.703 

.147 

5.095 

.000 

.436 

.190 

1-6 

.014 

.763 

.665 

.186 

6.724 

.000 

.470 

.221 

Avl 

.009 

.493 

..895 

.167 

5.897 

.000 

.449 

.201 

2-1 

.005 

.303 

.980 

.229 

8.482 

.000 

.490 

.240 

2-2 

.012 

.652 

.769 

.225 

8.396 

.000 

.495 

.245 

2-3 

.009 

.462 

.914 

.199 

7.156 

.000 

.468 

.219 

2-4 

.016 

.887 

.545 

.207 

7.727 

.000 

.498 

.248 

Av2 

.010 

.597 

,817 

.247 

9.488 

.000 

.518 

.268 

AVG 

.006 

.324 

.975 

.220 

8.192 

.000 

.496 

.246 

Notes:  Sec  text  for  explanation  of  Approach  1 .  Block  codes  defined  in  Figure  2. 
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The  data  in  Table  8  indicate  that  the  ASVAB  tests  added  nothing  to  the  LAMP  test  in 
predicting  the  development  of  procedural  skill.  This  was  to  be  expected,  since  the  ASVAB  tests  were 
designed  to  predict  technical  school  grades,  which  are  primarily  a  reflection  of  declarative  knowledge 
acquisition.  On  the  other  hand,  the  LAMP  tests  added  16.7%  unique  variance  to  the  ASVAB  tests  in 
predicting  average  latency  scores  on  gates  without  negations,  24.7%  in  predicting  the  average  on  gates 
with  negations,  and  22.0%  in  predicting  the  average  across  all  gate  blocks. 

Again,  these  estimates  arc  somewhat  inaccurate  because  of  the  varying  numbers  of 

independent  variables  included  the  equations.  Therefore,  an  Approach  2  analysis,  based  on  adjusted 
2 

R  s,  is  reported  in  Table  9. 


Tabic  9.  Unique  Contributions  of  ASVAB  and  LAMP  tests  to  the  Prediction  of  Study  1  Latency 
Criteria,  Estimated  by  Approach  2.  _ 


FULL  MODEL 
BLOCK  R  ADJ  R2 

LAMP 

R  ADJ  R2 

ASVAB 

R  ADJ  R2 

LAMP/ 

ASVAB 

ASVAB/ 

LAMP 

1-1 

.443 

.149 

.418 

.146 

.172 

.007 

.142 

.015 

1-2 

.372 

.087 

.359 

.099 

.173 

.008 

.079 

-.012 

1-3 

.379 

.093 

.364 

.103 

.199 

.018 

.075 

-.010 

1-4 

.428 

.135 

.Si9 

.147 

.172 

.008 

.127 

-.012 

1-5 

,436 

.142 

.420 

.147 

.208 

.021 

.121 

-.005 

1-6 

.470 

.174 

.454 

.179 

.185 

.012 

.162 

-.005 

Avl 

.449 

.154 

.438 

.164 

.184 

.012 

.142 

-.010 

2-1 

.490 

.195 

.455 

.208 

.105 

-.011 

.206 

-.013 

2-2 

.495 

.200 

.483 

.206 

.139 

-.003 

.203 

-.006 

2-3 

.468 

.173 

.459 

.183 

.144 

-.002 

.175 

-.010 

2-4 

.498 

.203 

.482 

.205 

.203 

.019 

.184 

.002 

Av2 

.518 

.225 

.508 

.232 

.147 

-.007 

.232 

-.007 

AVO 

.4% 

.201 

.490 

.213 

.169 

.004 

.197 

-.012 

Notes 

In  those  instances  in  which  the  adjusted  R2  for  the  LAMP  tests  is  higher  than  that  for  the 

FULL  MODEL,  the  estimated  contribution  for  the  ASVAB  tests  is  negative.  Block  codes  are  defined 
in  Figure  2.  See  text  for  explanation  of  Approach  2. 
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The  Approach  2  data  in  Table  9  tell  essentially  the  same  story.  The  ASVAB  tests  did  not 
contribute  to  the  LAMP  tests  in  accounting  for  the  latency  criteria.  As  a  matter  of  fact,  in  most 
instances  the  estimated  unique  contribution  of  ASVAB  tests  was  a  negative  value  (due  to  imperfections 
in  out  estimation  procedures)  which  occurred  when  the  adjusted  R2  for  the  LAMP  tests  was  higher 
than  that  for  the  FULL  MODEL.  On  the  other  hand,  the  LAMP  tests  added  substantially  to  ASVAB 
tests  in  predicting  these  criteria.  The  LAMP  contributions  were  estimated  to  be  14.2%  for  an  average 
across  gates  without  negations,  23.2%  for  an  average  across  gates  with  negations,  and  19.7%  across  all 
gate  blocks. 

As  mentioned  previously,  procedural  skill  in  this  study  was  reflected  in  the  time  required  for 
subjects  to  analyze  the  input  signals  of  logic  gates  and  correctly  determine  their  output  signals. 
However,  in  Block  1  some  individuals  made  a  number  of  errors  in  specifying  output  signals,  indicating 
that  their  understanding  of  the  rules  was  weak  at  that  stage.  The  meaning  of  latency  scores  is  unclear 
when  accuracy  is  low,  and  such  scores  are  likely  to  be  unpredictable  by  ability  factors.  In  a  sense,  they 
introduced  a  type  of  ‘error  variance’  in  the  criterion  scores.  This  problem  becomes  less  and  less 
pronounced  in  the  later  blocks,  in  which  most  responses  were  correct. 

One  approach  t  clarifying  latency  data  is  to  discard  subjects  who  do  not  achieve  some 
minimum  level  of  declarative  knowledge  during  the  course.  This  procedure  tends  to  purify  the  latency 
scores  as  a  skill  acquisition  criterion.  In  the  present  study,  subjects  were  eliminated  who  did  not 
achieve  at  least  a  90%  accuracy  level  in  the  fourth  block  of  gates  without  negations.  Thus,  104  subjects, 
were  eliminated,  leaving  344  subjects  in  the  residual  sample.  The  average  accuracy  levels  for  these  344 
subjects  across  the  six  blocks  of  gates  without  negations  was  81.8%,  90.6%,  94.2%,  96.4%,  96.6%,  and 
96.9%,  respectively,  and  86.5%,  90.0%,  92.3%,  and  92.5%  for  the  gates  with  negations,  respectively. 
Tables  10  and  11  report  data  comparable  to  that  reported  in  Tables  8  and  9,  except  that  they  are  based 
on  subjects  in  the  residual  sample. 

The  multiple  correlations  in  Table  10  arc  higher  than  they  were  in  Table  8,  since  there  was 
less  variance  associated  with  incorrect  responses  in  the  residual  sample  data.  However,  the  conclusions 
based  upon  these  results  arc  ihc  same.  The  ASVAB  tests  added  nothing  to  LAMP  tests  in  accounting 
for  procedural  skill  development  as  reflected  in  the  latency  data.  On  the  other  hand,  LAMP  tests 
added  considerable  variance  to  the  ASVAB  tests  in  accounting  for  procedural  skill  development.  The 
unique  contribution  of  LAMP  tests  was  estimated  to  be  22.8%  for  an  average  across  the  blocks  of  gates 
without  negations,  24.2%  for  an  average  across  blocks  with  negations,  and  24.0%  for  an  average  across 
all  blocks. 
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The  data  in  Table  11  represent  similar  findings,  except  that  the  estimated  LAMP  contribution 
levels  arc  slightly  lower,  being  16.5%  for  the  blocks  without  negations,  21.7%  for  the  blocks  with 
negations,  and  21.3%  across  all  blocks. 


Table  10.  Unique  Contributions  of  ASVAB  and  LAMP  Tests  to  the  Prediction  of  Study  1  Latency 
Criteria,  Estimated  by  Approach  1  in  the  Residual  Sample  (N=344) 

ASVAB  ADDS  TO  LAMP 

LAMP  ADDS  TO  ASVAB 

FULL-MODEL 

BLK 

R2 

F 

SIGN. 

R2 

F 

SIGN. 

R 

R2 

1-1 

.044 

1.815 

.057 

.193 

5.356 

.000 

.486 

.236 

1-2 

.032 

1.324 

.216 

.138 

3.756 

.000 

.469 

.220 

1-3 

.018 

.744 

.682 

.163 

4.555 

.000 

.489 

.239 

1-4 

.010 

.409 

.942 

.210 

6.005 

.000 

.508 

.258 

1-5 

.014 

.636 

.783 

.212 

6.29.3 

.000 

.534 

.285 

1-6 

.014 

.630 

.788 

.243 

7.298 

.000 

.542 

.239 

AvI 

.018 

.842 

.589 

.228 

6.939 

.000 

.550 

.303 

2-1 

.019 

.811 

.618 

.216 

6.286 

.000 

.522 

.272 

2-2 

.010 

.447 

.923 

.217 

6.402 

.(XX) 

.531 

.282 

2-3 

.011 

.467 

.911 

.193 

5.475 

.000 

.504 

.254 

2-4 

.020 

.927 

.509 

.131 

7.009 

.000 

.550 

.302 

Av2 

.017 

.798 

.630 

.242 

7.500 

.000 

.561 

.315 

AV(i 

.019 

.879 

.553 

.240 

7.375 

.000 

.558 

.311 

Notes:  See  text  for  explanation  of  Approach  1,  Block  codes  are  defined  in  Figure  2. 
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Table  11.  Unique  Contributions  of  ASVAB  and  LAMP  tests  to  the  Prediction  of  Study  1  Latency 
Criteria,  Estimated  by  Approach  2  in  the  Residual  Sample  (N  =  344) 


BLOCK 

FULL  MODEL 

R  ADJ  R2 

LAMP 

R 

ADJ  R2 

ASVAB 

R  ADJ  R2 

LAMP/ 

ASVAB 

ASVAB/ 

LAMP 

1-1 

.486 

.176 

.439 

.156 

.208 

.015 

.161 

.020 

1-2 

.469 

.159 

.433 

.150 

.286 

.082 

.077 

.009 

1-3 

.489 

.179 

.470 

.186 

.275 

.048 

.131 

-.007 

1-4 

.508 

.200 

.498 

.214 

.219 

.019 

.181 

-.014 

1-5 

.534 

.229 

.501 

.238 

.271 

.045 

.184 

-.014 

1-6 

.542 

.239 

.529 

.247 

.226 

.023 

.216 

-,(X)8 

Avl 

.550 

.212 

.533 

.252 

.273 

.047 

.165 

-.040 

2-1 

.522 

.215 

.503 

.220 

.238 

.028 

.187 

-.005 

2-2 

.531 

.226 

.521 

.239 

.255 

.037 

.188 

-.013 

2-3 

.504 

.1% 

.493 

.209 

.248 

.033 

.163 

-.013 

2-4 

.550 

.247 

.531 

.249 

.267 

.043 

..204 

-.002 

Av2 

.561 

.261 

.546 

.266 

.270 

.044 

.217 

-.005 

AVG 

.558 

.257 

.540 

.260 

.268 

.044 

.213 

.018 

Notes.  In  those  instances  in  which  the  adjusted  R2  for  the  LAMP  tests  is  higher  than  that  for  the 
FULL  MODEL,  the  estimated  contribution  for  the  ASVAB  tests  is  negative.  Block  codes  are  defined 
in  Figure  2.  See  text  for  explanation  of  Approach  2. 
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STUDY  2 


Study  2  involved  several  modifications  of  the  gates  course  and  in  the  LAMP  tests  selected  for 
validation.  To  provide  a  longer  practice  period  for  the  subjects  to  improve  their  procedural  skills,  the 
course  was  modified  to  include  eight  blocks  of  gates  without  negations  instead  of  only  six.  In  addition, 
complex  gate  circuit  problems  were  included  to  increase  job  relevance.  The  inclusion  of  more  learning 
blocks  reduced  the  time  available  for  predictor  test  administration,  so  the  Number  Fact  Retrieval, 
Meaning  Identity,  123  Number  Reduction,  and  Digit  Span  tests  were  dropped  in  order  to  increase  the 
number  of  non-quantitativc  tests  defining  the  working  memory  factor;  to  include  measures  of  spatial 
abilities  and  associative  learning  abilities,  four  new  tests  were  added:  General  Knowledge,  Word 
Association,  Figure  Recognition,  and  Mental  Character  Generation.  Finally,  a  pretest  of  ’going-in’ 
knowledge  was  administered  to  subjects.  This  pretest  was  made  up  from  problems  selected  from  the 
course  to  follow.  To  decrease  the  probability  of  subjects  recording  the  correct  answers  by  chance,  six 
incorrect  but  plausible  question  alternatives  (foils)  were  generated  and  included  as  possible  answers  to 
each  question.  Scores  from  this  pretest  were  at  chance  level  (M  =  14.6,  S.D.=  12.1),  and  they  did  not 
correlate  significantly  with  any  of  the  accuracy  or  latency  blocks  in  Study  2.  It  was  concluded  that  the 
course  represented  a  relatively  novel  learning  situation  to  the  subjects. 

Table  12  reports  the  mean  percent  correct  and  the  mean  latency  scores  for  the  eight  blocks  of 
simple  gates  without  negations  and  for  averages  across  blocks  in  other  sections  of  the  course.  The 
development  of  procedural  skill  is  clearly  reflected  in  the  systematic  reduction  of  response  times  across 
the  eight  blocks  of  simple  gates  without  negations;  the  mean  of  median  latencies  dropped  from  2.21 
seconds  in  Block  1  to  1.23  seconds  in  Blocks,  with  a  corresponding  drop  in  standard  deviations  from 
.69  seconds  in  Block  1  to  .32  seconds  in  Block  8. 

The  percent  correct  scores  also  reflect  a  systematic  strengthening  of  declarative  knowledge 
across  these  same  blocks,  with  the  means  increasing  from  83.7  in  Block  1  to  97.4  in  Block  8.  There  was 
an  increasing  restriction  in  range  due  to  ceiling  effects,  which  was  more  serious  than  that  encountered 
in  Study  1.  By  the  8th  block,  50%  of  the  subjects  made  perfect  scores,  and  another  29%  missed  only 
one  problem.  The  standard  deviation  of  the  scores  in  Block  8  was  only  4.0%.  The  effect  of  the  ceiling 
effect  was  to  render  the  later  blocks  less  valuable  as  measures  of  individual  differences  in  knowledge 
acquisition.  At  the  same  time,  since  most  subjects  made  perfect  or  near  perfect  scores  in  the  later 
blocks,  the  latency  scores  in  these  blocks  were  wcllsuited  as  indicators  of  their  level  of  procedural  skill. 
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Table  12. 

Means  and  Standard  Deviations  for 

Study  2  Blocks  And  Averages 

ACCURACY 

MEANS 

S.D.’s 

LATENCY 

MEANS 

S.D’s 

1-1 

83.67 

11.62 

2.21 

.69 

1-2 

89.15 

11.23 

1.83 

.55 

1-3 

92.66 

9.40 

1.63 

.56 

1-4 

95.02 

7.19 

1.46 

.48 

1-5 

95.86 

5.80 

1.34 

.42 

1-6 

96.12 

5.75 

1.32 

.38 

1-7 

96.83 

4.73 

1.25 

.36 

1-8 

97.35 

4.02 

1.23 

.32 

Avl 

93.33 

6.06 

1.46 

.38 

2-1 

87.25 

14.91 

3.33 

.80 

Av3 

91.47 

9.94 

2.30 

.61 

Av4 

89.13 

11.79 

3.47 

1.06 

AVG 

0151 

7.89 

2.01 

.46 

Notes.  Block  codes  arc  defined  in  Figure  2. 


Factor  Analysis 

In  a  factor  analysis  of  the  predictor  tests,  six  principal  axis  factors  were  extracted  and  rotated 
using  the  Varimax  method.  The  results  of  this  analysis  are  reported  in  Appendix  B. 

The  six  factors  were  given  the  same  names  as  the  six  factors  identified  in  Study  1.  They  were 
essentially  equivalent,  except  for  the  Working  Memory  factor,  which  was  now  defined  by  verbal  and 
spatial  tests  as  well  as  by  quantitative  tests,  and  for  the  Processing  Speed  factor,  which  was  a  little 
weaker  because  the  Meaning  Identity  test  had  been  dropped  from  the  battery.  Also,  the  Numerical 
Facility/Pcrceptual  Speed  factor  contained  less  numerical  variance  because  of  the  omission  of  the 
Number  Fact  Retrieval  test.  The  correlations  between  factor  scores  and  the  various  gates  course 
accuracy  criteria  are  presented  in  Table  13,  while  those  for  the  course  latency  criteria  are  presented  in 
Table  14. 
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Tabic  13.  Correlations  of  Factor  .Scores  with^tud^2^\ccurac^Criteria 


BLK 

VERBAL 

WKMEM 

PROCSPD 

PROCACC 

TECHKNOW 

NUM  PS 

1-1 

.15 

.53 

-.09 

.22 

.05 

.14 

1-2 

.12 

.50 

-.08 

.25 

.07 

.11 

1-3 

.04 

.50 

-.07 

.24 

.01 

.10 

1-4 

.06 

.45 

-.08 

.24 

.04 

.08 

1-5 

.01 

.37 

-.09 

.23 

.10 

.11 

1-0 

.03 

.31 

-.01 

.21 

-.01 

.09 

1-7 

-.01 

.28 

-.06 

.18 

-.04 

.09 

1-8 

-.02 

.26 

-.09 

.21 

-.08 

.12 

Avl 

.08 

.53 

-.09 

.28 

.03 

.13 

2-1 

.14 

.43 

-.05 

.22 

.17 

.12 

Av3 

.15 

.49 

-.11 

.28 

.12 

.05 

AVCi 

.15 

.54 

-.11 

.30 

.12 

.11 

Notes.  Block  codes  arc  defined  in  Figure  2. 


The  criteria  in  these,  as  well  as  in  later  tables,  include  eight  blocks  of  simple  gates  without 
negations,  an  average  across  these  eight  blocks,  a  single  block  of  simple  gates  with  negations,  an 
average  foi  the  two  blocks  of  mixed  simple  gates  (with  and  without  negations),  an  average  across  the 
three  blocks  of  mixed  simple  gates  and  complex  gate  circuits  (with  and  without  negations),  and,  finally, 
an  average  across  all  of  the  course  blocks. 

Data  in  Table  14  reveal  that  the  Working  Memory  factor  scores  had  highest  relationships  with 
all  of  the  accuracy  criterion  variables,  which  is  consistent  with  the  findings  in  Study  1.  It  should  be 
noted,  however,  that  the  magnitude  of  the  relationships  dropped  systematically  from  Block  1  to  Block 
8  of  simple  gates  without  negations.  This  downturn  was  expected,  and  is  attributed  to  the  decreasing 
individual  differences  variance  due  to  ceiling  effects.  The  correlation  of  the  Working  Memory  factor 
scores  with  an  average  across  all  blocks  of  simple  gates  without  negations  was  .530,  and  was  .543  for  an 
average  across  all  blocks  in  the  course. 
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Table  14. 

Correlations  of  Factor  Scores  with  Study  2 

Latency 

Criteria 

BLK 

VERBAL 

WKMEM 

PROCSPD 

PROCACC  TECHKNOW 

NUM  PS 

1-1- 

-.09 

.09 

.19 

.10 

-.06 

-.09 

1-2 

-.17 

-.14 

.28 

.02 

-.12 

-.14 

1-3 

-.15 

-.28 

.30 

-.03 

-.06 

-.12 

1-4 

-.11 

-.28 

.27 

-.05 

-.05 

-.10 

1-5 

-.12 

-.34 

.31 

-.07 

-.09 

-.07 

1-6 

-.12 

-.30 

.40 

-.06 

-.07 

-.10 

1-7 

-.14 

-.32 

.35 

-.04 

-.06 

-.14 

1-8 

-.16 

-.30 

.37 

-.02 

-.05 

-.12 

Avl 

-.15 

-.29 

.39 

-.02 

-.07 

-.13 

2-1 

-.09 

-.15 

.26 

.10 

-.11 

-.13 

Av3 

-.14 

-.11 

.35 

.11 

-.05 

-.15 

Av4 

-.08 

.09 

.21 

.14 

-.02 

-.21 

AVG 

-.16 

-.13 

.36 

.09 

-.Of) 

-.16 

Notes.  Block  codes  arc  defined  in  Figure  2. 


Again  consistent  with  the  findings  in  Study  1,  the  Processing  Accuracy  factor  scores  had  the 
second  highest  relationships  with  accuracy  scores  from  the  gates  course.  The  Numerical 
Facility/Perccptual  Speed,  Verbal  and  Technical  Knowledge  factor  scores  had  low  but  significant 
relationships  with  scattered  test  blocks,  correlating  -.110,  .145,  and  .117,  respectively,  with  the  average 
accuracy  across  all  course  blocks.  The  data  in  Table  14  show  that  the  Processing  Speed  factor  scores 
had  the  highest  relationships  with  the  latency  criteria,  again  consistent  with  the  findings  in  Study  1.  The 
Working  Memory  factor  scores  were  particularly  strong  in  predicting  latencies  in  the  later  blocks  of 
simple  gates  without  negations,  thus  replicating  another  finding  in  Study  1.  This  finding  is  consistent 
with  the  hypothesis  that  individuals  with  high  working-memory  capacity  attained  sufficient  strength  in 
declarative  knowledge  in  the  early  blocks  to  facilitate  their  development  of  procedural  skill.  The 
Numerical  Facility/Perccptual  Speed  and  the  Verbal  factor  scores  also  had  scattered  relationships  with 
latency  criteria,  attaining  correlations  of  -.160  and  -.155  respectively  with  the  average  latency  across  all 
blocks. 


In  summary,  the  results  of  the  factor  analysis  in  Study  2  were  remarkably  similar  to  those 
obtained  in  Study  1,  in  spite  of  changes  in  the  criterion  and  predictor  variables.  The  same  number  of 
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factors  was  obtained  in  both  studies,  and  the  patterns  of  factor-criterion  relationships  were  highly 
similar. 

Comparative  Validities  of  ASVAB  and  LAMP  Tt.  for  Gates-Course  Learning  Criteria 
Actiwisition  of  Declarative  Knowledge 

As  was  the  case  in  Study  1,  the  accuracy  scores  from  the  gatcs-course  test  blocks  were 
accepted  as  indicators  of  acquired  declarative  knowledge.  The  same  two  approaches  were  taken  to 
determine  the  ability  of  LAMP  tests  alone,  ASVAB  tests  alone,  and  a  combination  of  LAMP  and 
ASVAB  tests  to  predict  these  scores.  In  Approach  1,  the  losses  in  R2s  were  determined  as  LAMP  tests 
and  ASVAB  tests  were  removed  one  set-at-a-time  from  full  models  containing  both  classes  of 
predictors.  In  each  instance,  the  unique  contribution  of  the  two  predictor  sets  were  computed,  along 
with  F-ratios  and  significance  levels  based  on  appropriate  degrees  of  freedom.  The  results  from  this 
first  analysis  are  reported  in  Table  15. 


Table  15.  Unique  Contributions  of  ASVAB  and  LAMP  Tests  to  the  Prediction  of  Study  2  Accuracy 
Criteria,  Estimated  by  Approach  1 

ASVAB  ADDS  TO  LAMP  LAMP  ADDS  TO  ASVAB  FULL-MODEL 
BLK  R2  F  SIGN.  R2  F  SIGN.  R  R2 


1-1 

.043 

2.764 

.003 

.155 

7.166 

.000 

.611 

.373 

1-2 

.039 

2.376 

.010 

.146 

6.428 

.000 

.585 

.342 

1-3 

.051 

3.047 

.001 

.158 

6.683 

.000 

.561 

.315 

1-4 

.042 

2.394 

.009 

.152 

6.214 

.000 

.541 

.293 

1-5 

.022 

1.163 

.314 

.127 

4.742 

.000 

.474 

.224 

1-6 

.025 

1.243 

.261 

.105 

3.699 

.000 

.422 

.178 

1-7 

.028 

1.372 

.191 

.091 

3.163 

.(XX) 

.404 

.163 

1-8 

.039 

1.921 

.041 

.094 

3.315 

.000 

.418 

.175 

Avl 

.042 

2.770 

.003 

.189 

8.929 

.000 

.622 

.387 

2-1 

.053 

3.120 

.000 

.120 

5.017 

.000 

.553 

.305 

Av3 

.055 

3.310 

.000 

.136 

5.901 

.000 

.575 

.330 

Av4 

.016 

1.035 

.413 

.240 

11.432 

.000 

.625 

.391 

AVG 

.031 

2.215 

.016 

.222 

11.393 

.(XX) 

.659 

.434 

Notes:  Block  codes  defined  in  Figure  2.  See  text  for  explanation  of  Approach  1. 
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In  Approach  2,  multiple  correlations  were  computed  for  the  LAMP  tests  alone,  the  ASVAB 

tests  alone,  and  for  a  combination  of  LAMP  and  ASVAB  tests  against  the  various  accuracy  criteria. 

2 

The  unique  contribution  of  each  battery  to  the  other  was  estimated  by  the  differences  in  R  s  computed 
i  the  restricted  and  full  models,  alter  such  R2s  had  been  adjusted  for  the  number  of  independent 
variables  to  be  more  accurate  estimates  of  population  parameters.  The  results  of  this  second  analysis 
are  reported  in  Table  16. 


Table  16.  Unique  Contributions  of  ASVAB  and  LAMP  tests  to  the  Prcdiciton  of  Study  2  Accuracy 
Criteria,  Estimated  by  Approach  2 

BLOCK 

FULL  MODEL 

R  ADJ  R2 

LAMP 

R  ADJ  R2 

ASVAB 

R  ADJ  R2 

LAMP/ 

ASVAB 

ASVAB/ 

LAMP 

1-1 

.611 

.336 

.575 

.308 

.476 

.200 

.136 

.028 

1-2 

.585 

.303 

.551 

.280 

.443 

.177 

.126 

.023 

1-3 

.561 

.274 

.513 

.238 

.3% 

.136 

.138 

,0V) 

1-4 

.541 

.151 

.501 

.226 

.376 

.121 

.130 

.025 

1-5 

.474 

.179 

.450 

.175 

.312 

.076 

.103 

.004 

1-6 

.422 

.130 

.391 

.124 

.271 

.051 

.079 

,(X)6 

1-7 

.404 

.114 

.376 

.106 

.268 

.050 

.064 

.008 

1-8 

.418 

.126 

.369 

.107 

.284 

.059 

.089 

.019 

Avl 

.622 

.351 

.588 

.324 

.446 

.180 

.171 

.027 

2-1 

.553 

.264 

.502 

.227 

.430 

.166 

.098 

.037 

Av3 

.575 

.291 

.525 

.251 

.441 

.175 

.116 

.040 

Av4 

.625 

.355 

.612 

.354 

.388 

.130 

.225 

.001 

AVG 

.659 

.400 

.635 

.383 

.460 

.193 

.207 

.017 

Notes.  Block  codes  arc  defined  in  Figure  2.  See  text  for  explanation  of  Approach  2. 


The  impact  of  the  ceiling  effects  on  the  eight  blocks  of  simple  gates  without  negations  is 
reflected  in  Tables  15  and  16  by  a  systematic  reduction  in  the  magnitudes  of  the  full-model  Rs  from 
.611  in  Block  1  to  .418  in  Block  8.  The  unique  contribution  of  ASVAB  to  LAMP  tests  in  the  prediction 
of  these  criteria  was  small,  and  in  some  instances  not  significant.  On  the  other  hand,  the  LAMP  tests 
contributed  significantly  to  the  ASVAB  tests  in  predicting  all  blocks  in  the  course,  including  the  blocks 
of  simple  gates.  In  spite  of  the  ceiling  effects,  the  estimated  contribution  of  LAMP  to  ASVAB  tests  in 
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predicting  average  accuracy  across  the  first  eight  blocks  was  estimated  to  be  18.9%  by  Approach  1 
(Table  15)  and  17.1%  by  Approach  2  (Tabic  16).  The  LAMP  tests  were  particularly  powerful  in 
predicting  average  accuracy  scores  across  the  blocks  which  contained  complex  gate  circuits,  yielding  a 
multiple  correlation  of  .612  compared  with  .388  for  the  ASVAB  tests.  In  this  instance,  the  unique 
contribution  of  the  LAMP  tests  to  ASVAB  was  estimated  to  be  24.0%  by  Approach  1,  and  22.5%  by 
Approach  2.  The  unique  contribution  of  ASVAB  tests  to  LAMP  tests  was  estimated  to  be  non¬ 
significant  (F-ratio  =  1.035,  P>  .4).  The  multiple  correlation  of  the  LAMP  tests  with  average  accuracy 
across  the  entire  course  was  .635,  compared  with  .460  for  the  ASVAB  tests,  and  the  uiiiquc 
contribution  of  LAMP  tests  to  ASVAB  was  estimated  to  be  22.2%  by  Approach  1  and  20.7%  by 
Approach  2.  These  estimates  are  to  be  compared  with  the  3.1%  and  1.7%  estimates  of  the 
contribution  of  ASVAB  tests  to  LAMP  tests,  which  were  not  significant  at  the  .01  level.  All  of  these 
values  are  remarkably  similar  to  those  obtained  in  Study  1. 


Acquisition  of  Procedural  Skill 

As  in  Study  1,  latency  scores  were  used  as  indicants  of  procedural  skill.  Once  individuals  had 
the  rules  for  solving  gate  problems  firmly  in  mind,  they  tended  to  accurately  apply  them  with  increasing 
speed  and  efficiency.  For  most  individuals,  proccduraiization  of  rules  began  even  before  they  reached 
the  Block  1  problem  set.  Some  individuals  made  perfect  scores  on  all  problems  in  this  block,  and 
solved  each  problem  with  increased  speed.  Others  were  relatively  accurate  on  AND  or  OR  gates,  but 
were  still  somewhat  confused  and  inaccurate  in  solving  NOR  gales.  The  increase  in  accuracy  from 
Block  1  to  Block  8  reflected  the  continued  acquisition  and  strengthening  of  declarative  knowledge 
(concerning  gate  rules)  within  the  subject  sample,  and  by  the  8th  block,  a  97%  accuracy  level  was 
obtained.  Latency  scores  were  the  best  measures  of  individual  differences  in  procedural  skill  at  any 
learning  stage,  but  this  criterion  was  contaminated  to  the  extent  that  declarative  learning  had  not  been 
achieved.  For  this  reason,  the  latency  scores  in  Study  2  were  analyzed  in  the  same  manner  as  they  were 
in  Study  1.  First,  they  were  analyzed  in  the  full  sample,  using  the  same  two  approaches  as  were  used 
for  analyzing  accuracy  scores.  The  results  of  these  analyses  are  reported  in  Tables  17  and  18.  Then 
they  were  analyzed  on  a  subsample  of  individuals  who  reached  the  90%  accuracy  level  by  the  4th  block 
of  simple  gates  without  negations.  The  results  of  these  analyses  are  presented  in  Tables  19  and  20. 
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Table  17.  Unique  Contributions  of  ASVAB  and  LAMP  Tests  to  the  Prediction  of  Study  2  Latency 
Criteria,  Estimated  by  Approach  1 

ASVAB  ADDS  TO  LAMP  LAMP  ADDS  TO  ASVAB  FULL-MODEL 
BLK  R2  F  SIGN.  R2  F  SIGN.  R  R2 


1-1 

.039 

1.867 

.048 

.097 

3.272 

.000 

.380 

.145 

1-2 

.024 

1.202 

.288 

.131 

4.754 

.000 

.450 

.203 

1-3 

.014 

.744 

.673 

.152 

5.821 

.000 

.490 

.240 

1-4 

.024 

1.231 

.269 

.135 

4.950 

.000 

.457 

.209 

1-5 

.028 

1.607 

.102 

.194 

7.842 

.000 

.5.30 

.281 

1-6 

.022 

1.251 

.256 

.213 

8.823 

.(XX) 

.547 

.301 

1-7 

.012 

.668 

.754 

.172 

6.827 

.000 

.517 

.267 

1-8 

.011 

.620 

.7% 

.182 

7.261 

.000 

.522 

272 

Avl 

.019 

1.001 

.361 

.205 

8.421 

.000 

.542 

.294 

2-1 

.025 

1.249 

.258 

.143 

5.188 

.000 

.448 

.201 

Av3 

.022 

1.169 

.310 

.171 

6.486 

.000 

.487 

.237 

Av4 

.017 

.945 

.491 

.214 

8.334 

.(XX) 

.505 

.256 

AVG 

.020 

1.113 

.350 

.206 

8.260 

.000 

.526 

.277 

Notes:  Block  codes  defined  in  Figure  2.  See  text  for  explanation  of  Approach  1. 


The  results  reported  in  Tables  17  and  18  are  comparable  to  those  obtained  in  Study  1.  In  no 
instance  did  the  ASVAB  tests  add  unique  valid  variance  (at  the  .01  level)  to  the  LAMP  tests  in 
predicting  latency  scores.  On  the  other  hand,  the  unique  contribution  of  LAMP  tests  to  ASVAB  in 
predicting  the  average  latency  on  simple  gates  without  negations  was  estimated  by  the  two  approaches 
to  be  20.5%  and  18.4%,  respectively.  The  multiple  correlation  of  LAMP  tests  for  the  average  latency 
scores  across  all  gate  blocks  was  .507,  compared  with  .267  for  ASVAB  tests,  and  the  Approach  1-. 
Approach  2  estimates  of  the  unique  contribution  of  LAMP  tests  to  ASVAB  tests  in  predicting  this 
criterion  were  20.6%  and  18.5%,  respectively. 
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Table  Ifi.  Unique  Contributions  of  ASVAB  and  LAMP  tests  to  the  Prediction  of  Study  2  Latency 
Criteria,  Estimated  by  Approach  2 


BLOCK 

FULL  MODEL 

R  ADJ  R2 

R 

LAMP 

ADJ  R2 

ASVAB 

R  ADJ  R2 

LAMP/ 

ASVAB 

ASVAB/ 

LAMP 

1-1 

.380 

.094 

.324 

.075 

.219 

.025 

.069 

.019 

1-2 

.450 

.156 

.423 

.151 

.268 

.050 

.106 

.005 

1-3 

.490 

.196 

.476 

.200 

.297 

.066 

.130 

-.004 

1-4 

.457 

.162 

.430 

.157 

.272 

.052 

.110 

.005 

1-5 

.530 

.239 

.503 

.228 

.295 

.065 

.174 

.011 

1-6 

.549 

.260 

.529 

.255 

.297 

.067 

.193 

.005 

1-7 

.517 

.224 

.505 

.230 

.308 

.073 

.151 

-.006 

1-8 

.522 

.229 

.511 

.236 

.300 

.069 

.160 

-.007 

Avl 

.542 

.252 

.524 

.251 

.298 

.068 

.184 

.001 

2-1 

.448 

.154 

.420 

.149 

.241 

.036 

.118 

.005 

Av3 

.487 

.192 

.464 

.189 

.259 

.045 

.147 

.003 

Av4 

.505 

.212 

.488 

.213 

.205 

.019 

.167 

-.001 

AVO 

.526 

.234 

.507 

.232 

.267 

.049 

.185 

.002 

Notes.  In  those  instances  in  which  the  adjusted  R  2  for  the  LAMP  tests  is  higher  than  that  for  the 
FULL  MODEL,  the  estimated  contribution  for  the  ASVAB  tests  is  negative.  Block  codes  are  defined 
in  Figure  2.  Sec  text  for  explanation  of  Approach  2. 


Discarding  subjects  who  did  not  reach  the  90%  accuracy  level  by  the  4th  block  of  simple  gates 
without  negations  resulted  in  a  loss  of  only  63  subjects,  or  14.6%  of  the  original  sample.  However,  this 
smaller  sample  resulted  in  criterion  measures  which  were  considerably  less  contaminated  by  inaccurate 
responses.  By  the  4th  block  of  simple  gates  without  negations,  the  average  percent  correct  score  was 
over  97%,  and  this  average  rose  to  above  the  98%  level  by  the  8th  block.  Furthermore,  in  none  of 
blocks  of  the  more  complex  gates,  which  followed,  did  the  average  accuracy  level  fall  below  the  90% 
level. 


As  expected,  the  full-model  multiple  correlations  were  higher  in  the  residual  sample,  rising 
from  .404  in  Block  1  of  simple  gates  without  negations  to  .549  in  Block  8.  The  full-model  multiple 
correlation  was  .569  for  an  average  across  the  eight  blocks  without  negations,  and  .572  for  an  average 
across  all  course  blocks.  Again,  the  data  in  Tables  19  and  20  indicate  that  the  ASVAB  tests  did  not  add 
significantly  to  the  LAMP  tests  in  predicting  any  of  the  latency  criteria.  On  the  other  hand,  using  the 
two  approaches  previous  described,  the  estimated  contribution  of  LAMP  tests  to  ASVAB  tests  was 
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21.8%  and  19.6%  for  predicting  the  average  latency  across  simple  gates  without  negations,  and  21.9% 
and  19.7%  for  the  average  latency  across  all  course  blocks. 


Table  19.  Unique  Contributions  of  ASVAB  and  LAMP  Tests  to  the  Prediction  of  .Study  2  Latency 
Criteria,  Estimated  by  Approach  1  in  the  Residual  Sample  (N =368) 

ASVAB  ADDS  TO  LAMP  LAMP  ADDS  TO  ASVAB  FU  LL-MODEL 

BLK  R2  F  SIGN.  R2  F  SIGN.  R  R2 


1-1 

.036 

1.505 

.136 

.111 

3.257 

000 

.404 

.163 

1-2 

.028 

1.312 

.222 

.148 

4.908 

.000 

.511 

.261 

1-3 

.027 

1.282 

.239 

.159 

5.384 

.000 

.524 

.274 

1-4 

.031 

1.425 

.167 

.162 

5.360 

.000 

.508 

.258 

1-5 

.026 

1.291 

.234 

.204 

7.111 

.000 

.547 

.299 

1-6 

.027 

1.341 

.207 

.206 

7.266 

.000 

.553 

.306 

1-7 

.011 

.499 

.890 

.182 

6.137 

.000 

.525 

.275 

1-8 

.012 

.567 

.841 

.194 

6.803 

.000 

.549 

.302 

Avl 

.023 

1.149 

.325 

.218 

7.893 

.000 

.569 

.324 

2-1 

.019 

.910 

.524 

.198 

6.774 

.000 

.530 

.281 

Av3 

.017 

.874 

.558 

.211 

7.735 

.000 

.577 

.333 

Av4 

.019 

.892 

.541 

.188 

6.433 

.000 

.531 

.282 

AVG 

.018 

.908 

.525 

.219 

7.953 

.000 

.572 

.327 

Notes:  Block  codes  arc  defined  in  Figure  2.  Sec  text  for  explanation  of  Approach  1. 


Discussion 


All  in  all,  the  results  from  Study  2  were  very  similar  to  those  obtained  in  Study  1 .  The  ASVAB 
tests  were  found  to  be  poor  predictors  of  latency  scores  in  both  studies,  and  they  added  nothing  to  the 
LAMP  tests  in  predicting  such  criteria.  This  result  was  not  unexpected,  since  the  ASVAB  was  designed 
to  predict  technical  school  grades  and  has  historically  been  validated  against  grade  criteria.  On  the 
other  hand  the  LAMP  tests  produced  a  multiple  correlation  of  .490  in  Study  1  and  .507  in  Study  2  in 
predicting  the  average  latency  scores  in  the  full  samples.  When  the  samples  were  rcsiduali/cd  to 
include  only  those  subjects  who  had  accuracy  scores  of  at  least  90%  on  Block  4  of  simple  gates  without 
negations,  the  multiple  correlations  were  computed  to  be  .558  in  Study  1  and  .572  in  study  2. 
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Table  20.  Unique  Contributions  of  ASVAB  and  LAMP  tests  to  the  Prediction  of  Study  2  Latency 
Criteria,  Estimated  by  Approach  2  in  the  Residual  Sample  (N  =  368) 


BLOCK 

FULL  MODEL 

R  ADJ  R2 

LAMP 

R  ADJ  R2 

ASVAB 

R  ADJ  R2 

LAMP/ 

ASVAB 

ASVAB/ 

LAMP 

1-1 

.404 

.105 

.356 

.092 

.229 

.026 

.079 

.013 

1-2 

.511 

.209 

.482 

.202 

.336 

.088 

.077 

.007 

1-3 

.524 

.224 

.497 

.217 

.339 

.090 

.134 

.007 

1-4 

.508 

.206 

.476 

.196 

.309 

.070 

.136 

.010 

1-5 

.547 

.250 

.522 

.244 

.309 

.070 

.179 

.006 

1-6 

.553 

.258 

.528 

.250 

.317 

.075 

.183 

.008 

1-7 

.525 

.224 

.514 

.235 

.306 

.068 

.156 

-.011 

1-8 

.549 

.253 

.539 

.262 

.329 

.083 

.170 

-.009 

Avl 

.569 

.277 

.549 

.274 

.326 

.081 

.196 

.003 

2-1 

.530 

.231 

.512 

.232 

.288 

.057 

.174 

-.001 

Av3 

.577 

.286 

.562 

.289 

.350 

.098 

.188 

-.063 

Av4 

.531 

.232 

.513 

.234 

.?% 

.068 

.164 

-.004 

AVG 

.572 

.280 

.556 

.282 

9>9 

.083 

.197 

-.002 

Notes.  In  those  instances  in  which  the  R2  for  the  LAMP  tests  is  higher  than  that  for  the  FULL 
MODEL,  the  estimated  contribution  for  the  ASVAB  tests  is  negative.  Block  codes  are  defined  in 
Figure  2.  Sec  text  for  explanation  of  Approach  2. 


The  multiple  correlations  of  LAMP  tests  for  the  average  accuracy  scores  across  all  blocks  was 
.681  in  Study  1  and  .659  in  Study  2.  The  comparable  values  obtained  with  ASVAB  tests  were  .478  and 
.460,  respectively.  To  put  it  another  way,  the  LAMP  tests  accounted  for  more  than  twice  the  amount  of 
criterion  variance  as  was  accounted  for  by  the  ASVAB  tests.  The  unique  contribution  of  LAMP  tests 
to  ASVAB  tests  in  predicting  the  overall  accuracy  criteria  was  estimated  to  be  at  least  20%  in  both 
studies,  using  either  of  the  described  estimation  approaches. 


35 


Decision  on  Need  for  Study  3 


There  arc  two  factors  which  should  be  considered  before  accepting  the  findings  in  Studies  1 
and  2.  One  deals  with  common-methods,  and  the  other  with  concurrence.  In  the  present  studies,  the 
LAMP  tests  and  the  gates  course  criterion  tests  were  both  presented  by  microcomputers,  while  the 
ASVAB  tests  were  administered  using  papcr-and-pcncil  procedures.  The  common  methods  may  have 
given  an  advantage  to  the  LAMP  tests.  This  matter  will  be  considered  in  the  General  Discussion 
section.  The  other  factor  concerns  the  fact  that  the  LAMP  tests  and  the  course  criterion  tests  were 
administered  concurrency  (in  the  same  session),  while  the  ASVAB  tests  had  been  administered  some 
weeks  or  months  previously  in  the  operational  testing  stations.  It  is  reasonable  to  expect  that  abilities 
measured  at  the  time  of  course  administration  would  be  more  valid  predictors  of  course  outcome  than 
would  be  abilities  measured  at  some  prior  point  in  time.  This  matter  has  been  discussed  and  evaluated 
in  at  least  two  prior  studies.  Christa!  (1989)  conducted  a  study  based  on  several  thousand  subjects  and 
found  a  small  but  significant  concurrency  effect.  However,  this  study  used  ASVAB  tests  both  as 
predictors  and  criteria.  Divigi  (1990)  conducted  a  study  to  evaluate  the  incremental  validity  added  by 
experimental  te»is  to  ASVAB  in  predicting  job  knowledge  and  hands-on-performance  criteria  in  four 
occupations.  In  this  study  he  used  original  ASVAB  scores  and  scores  from  a  version  of  the  ASVAB 
administered  concurrently  with  the  experimental  tests.  He  found  the  difference  between  incremental 
variance  using  concurrent  and  enlistment  ASVABs  to  be  "minor  compared  to  variations  across 
different  criterion  variables  and  occupational  specialities." 

In  spite  of  the  evidence  that  concurrency  effects  were  not  likely  to  be  large,  a  decision  was 
made  to  conduct  a  third  study  to  specifically  test  for  them  in  the  context  of  validating  ASVAB  tests 
against  criteria  from  the  logic  gates  courses. 


S'i  UDY  3 

With  only  three  hours  of  testing  time  available,  it  was  not  possible  to  rcadminister  the  entire 
ASVAB  along  with  a  gates  course.  However,  results  from  the  factor  analyses  in  both  Study  1  and  in 
Study  2  indicated  that  most  of  the  predictive  validity  for  the  accuracy  criteria  came  from  measures  of 
working  memory.  Since  Arithmetic  Reasoning,  Mathematics  Knowledge,  and  Mechanical 
Comprehension  were  the  only  ASVAB  tests  with  significant  loadings  on  this  factor,  analyses  were 
conducted  using  Study  1  and  Study  2  data  to  compare  the  multiple  correlations  of  these  three  tests  with 
those  from  the  entire  ASVAB  in  predicting  the  various  accuracy  criteria.  The  results  from  these 
analyses  are  presented  in  Table  21. 
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Tabic  21.  Comparative  Multiple  Correlations  of  the  Entire  ASVAB  and  a  Subset  of  ASVAB  Tests  for 


Accuracy  Criteria  in  Study  1  and  Study  2 


ASVAB 

STUDY  1 

SUBSET* 

ASVAB 

STUDY  2 

SUBSET* 

1-1 

.381 

.381 

.460 

.452 

1-2 

.359 

.359 

.424 

.424 

1-3 

.315 

.315 

.376 

.359 

1-4 

.295 

.295 

.363 

.363 

1-5 

.310 

.297 

.273 

.273 

1-6 

.255 

.255 

.238 

.238 

Avl 

.255 

.255 

1-7 

.215 

.215 

1-8 

.264 

.243 

Avl 

.431 

.431 

2-1 

.447 

.429 

.415 

.415 

2-2 

.388 

.378 

2-3 

.391 

.381 

2-4 

.352 

.352 

Av2 

.433 

.421 

Avg3 

.428 

.416 

Avg4 

.360 

.360 

AV(  i 

.464 

.460 

.455 

.455 

Notes.  *  Subset  of  ASVAB  tests  is  composed  of  Arithmetic  Reasoning,  Mathematics  Knowledge,  and 


Mechanical  Comprehension.  Block  codes  are  defined  in  Figun-  2. 
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The  data  in  Table  21  confirm  that  essentially  all  of  the  criterion  variance  predicted  by  ASVAB 
could  be  captured  by  the  three  tests  mentioned;  therefore  they  were  administered  along  with  the  Study 
2  gates  course  to  312  airmen  in  the  11th  day  of  basic  military  training.  In  order  to  gauge  the  magnitude 
of  concurrency  effects,  multiple  correlations  for  each  of  the  course  accuracy  criteria  were  computed 
twice;  once  using  scores  on  the  three  tests  obtained  from  the  operational  testing  files,  and  again,  using 
scores  on  the  three  tests  from  the  concurrent  (retest)  administration.  The  results  of  these  analyses  are 
presented  in  Table  22. 

* 

Table  22.  Predictive  and  Concurrent  Multiple  Correlations  of  a  Subset  of  ASVAB  tests  for  Study  2 
Accuracy  Criteria 


BLK 

PREDICTIVE 

CONCURRENT 

1-1 

.456 

.456 

1-2 

.478 

.458 

1-3 

.455 

.453 

1-4 

.362 

.334 

1-5 

.338 

.309 

1-6 

.290 

.284 

1-7 

.251 

.249 

1-8 

.206 

.249 

Avl 

.453 

.433 

2-1 

.523 

.514 

Av3 

.498 

.431 

Av4 

.495 

.477 

AVG 

.534 

.505 

Notes.  *The  subset  of  ASVAB  tests  was  composed  of  Arithmetic  Reasoning,  Mathematics  Knowledge, 
and  Mechanical  Comprehension.  Block  codes  are  defined  in  Figure  2. 

The  data  in  Table  22  reveal  little  or  no  differences  in  the  multiple  correlations  based  on  the 
original  and  concurrent  test  scores.  Where  differences  did  exist,  they  tended  to  be  in  the  direction  of 
higher  validities  for  the  original  test  scores.  There  are  several  possible  explanations  for  this  unexpected 
finding  which  will  be  addressed  in  the  General  Discussion  section.  However,  the  present  results 
suggest  that  the  concurrency  effects  were  small,  if  they  existed  at  all. 
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CORRECTIONS  FOR  RESTRICTIONS  OF  RANGE 


Description  of  the  Problem 

AH  of  the  analyses  reported  thus  far  in  this  paper  were  based  upon  data  collected  from 
samples  of  airmen  in  Basic  Military  Training.  Individuals  in  these  samples  were  accepted  into  the  U.S. 
Air  Force  from  a  general  applicant  population  only  after  having  met  certain  minimum  aptitude 
requirement  levels  on  ASVAB  Aptitude  Indexes  (AI’s).  This  selection  process  resulted  in  a  restriction 
of  range  in  the  ASVAB  test  score  distributions,  which,  in  turn,  yielded  lower  ASVAB  test  validities 
than  would  have  been  obtained  in  the  full  applicant  population.  There  was  also  a  restriction  of  range  in 
the  LAMP  test-score  distributions,  to  the  extent  that  the  LAMP  tests  correlated  with  the  selection 
criteria.  Finally,  the  differential  restrictions  of  range  due  to  selection  resulted  in  overestimations  of 
what  the  unique  contribution  of  LAMP  tests  to  ASVAB  tests  would  be  in  the  applicant  population. 


Approach 

A  decision  was  made  to  correct  all  of  the  intcrcorrelations  and  validities  for  range  restrictions. 
However,  a  question  arose.  Which  of  three  populations  should  serve  as  the  basis  for  such  corrections? 
The  most  stable  would  be  the  18-23  year-old  youth  population  for  1980,  which  was  used  for  establishing 
the  ASVAB  norms.  However,  in  my  judgment,  use  of  this  population  would  not  provide  the  most 
meaningful  indication  of  the  utility  of  the  ASVAB  in  making  selection  decisions  for  an  all-volunteer 
force,  nor  would  it  provide  the  most  meaningful  indication  of  the  potential  contribution  of  the  LAMP 
tests  to  the  ASVAB  tests  in  making  such  decisions.  The  value  of  mental  tests  in  maintaining  an  all- 
volunteer  force  should  be  gauged  in  terms  of  their  power  to  identify  the  best-qualified  individuals  from 
among  those  being  considered  for  selection.  This  being  the  case,  the  value  of  tests  to  the  Department 
of  Defense  should  be  gauged  in  terms  of  their  ability  to  identify  the  best  qualified  from  among  those 
applying  for  entry  into  the  Uniformed  Services;  while  the  value  of  the  tests  to  the  U.S.  Air  Force  should 
be  gauged  in  terms  of  their  ability  to  identify  the  best  qualified  individuals  from  among  those  applying 
for  entry  into  the  U.S.  Air  Force. 

It  is  recognized  that  the  nature  of  applicant  populations  changes  over  time;  therefore,  they 
would  not  be  appropriate  for  establishing  operational  test  norms.  However,  they  can  be  used  to 
estimate  the  value  of  the  tests  in  selecting  the  best  qualified  from  those  being  considered  across  a  given 
time  period.  In  the  present  study,  a  method  suggested  by  Mifflin  &  Verna  (1977)  was  applied  to 
correct  the  test  validities  and  intcrcorrelations  to  a  random  sample  of  10,853  USAF  individuals  who 
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applied  for  entry  into  the  Air  Force  during  1980-1989.  Results  from  analyses  of  these  data  are  judged 
to  be  the  best  indication  of  the  contribution  of  LAMP  tests  to  ASVAB  tests  in  making  selection 
decisions  in  an  ali-voluntccr  force  environment.  Corrections  would  have  been  made  to  the  DOD 
applicant  population  during  this  same  time-period,  but  unfortunately,  data  were  not  available  when  this 
report  was  being  prepared. 

Data  were  available  for  approximately  10,000  subjects  designed  to  be  representative  of  the 
25,408,193-case  1980  youth  population  described  by  Maicr  &  Sims  (1986).  Corrections  were  made  to 
this  sample  to  get  a  rough  indication  of  test  utility  in  case  of  an  all-out  draft.  However,  in  my 
judgment,  use  of  this  sample  leads  to  an  overcorrection.  In  the  event  of  an  all-out  draft,  a  portion  of 
those  individuals  in  the  upper-aptitude  ranges  would  be  siphoned  off  into  officer  training  programs, 
and  some  in  critical  jobs  or  in  college  might  be  given  deferments.  Undoubtedly  there  would  also  be 
some  individuals  who  arc  physically  or  mentally  handicapped,  or  who  have  criminal  records,  who  would 
be  rejected  prior  to  psychological  testing.  These  factors  lead  one  to  predict  that  the  range  of  aptitudes 
in  the  sample  considered  for  draft  into  the  enlisted  ranks  would  be  less  than  the  range  of  aptitudes  in 
the  1980  youth  population.  In  spite  of  these  concerns,  corrections  to  this  population  were  made. 


Results 


Tables  23  reports  validities  of  the  ASVAB  and  LAMP  tests  for  the  Study  1  gates  course 
accuracy  and  latency  criteria  in  the  computing  sample,  as  well  as  those  corrected  to  the  USAF 
applicant  sample  and  the  1980  youth  sample.  Table  24  reports  the  same  information  based  on  Study  2 
data.  Table  25  reports  data  relating  to  the  contribution  of  LAMP  tests  to  ASVAB  tests  in  predicting 
Study  1  gates  course  accuracy  and  latency  criteria  in  the  USAF  applicant  sample,  while  Table  26 
reports  these  estimated  contributions  in  the  1980  youth  sample.  Tables  27  and  28  are  comparable  to 
Tables  25  and  26,  except  that  they  are  based  upon  Study  2  data. 

Table  29  presents  a  summary  of  LAMP  test  contributions  to  ASVAB  in  accounting  for  Study  1 
and  Study  2  accuracy  and  latency  criterion  variance  in  the  computing  sample,  in  the  USAF  applicant 
sample,  and  in  the  1980  youth  sample.  These  contributions  arc  impressive,  even  when  the  estimates 
are  made  using  differences  in  adjusted  R2s  which  have  been  corrected  to  the  1980  youth  sample. 
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Tabic  23.  Validities  of  ASVAB  and  LAMP  Tests  for  Study  1  Accuracy  and  Latency  Criteria  in  the 
Computing  Sample,  the  Air  Force  Applicant  Sample  and  the  1980  Youth  Sample 


ACCURACY _  _ LATENCY 


Test 

CS 

AAS 

YS 

CS 

AAS 

YS 

ASVAB 

GS 

.13 

.36 

.52 

-.01 

-.13 

-.22 

AR 

.40 

.54 

.65 

-.06 

-.16 

-.25 

WK 

.20 

.39 

.54 

-.03 

-.14 

-.24 

PC 

.16 

.38 

.51 

-.03 

-.14 

-.24 

NO 

.08 

.24 

.44 

-.13 

-.21 

-.29 

CS 

.09 

.22 

.40 

-.10 

-.17 

-.25 

AS 

.10 

.28 

.39 

.01 

-.05 

-.13 

MK 

.40 

.51 

.63 

-.06 

-.15 

-.24 

MC 

.29 

.43 

.53 

.03 

-.06 

-.15 

El 

.26 

.38 

.53 

.05 

-.03 

-.14 

LAMP 

CRA 

.27 

.29 

.32 

.05 

.03 

-.00 

NIA 

.29 

.32 

.36 

.07 

.04 

-.01 

PIA 

.27 

.27 

.28 

.08 

.07 

.05 

DSA 

.32 

.41 

.50 

-.07 

-.12 

-.19 

NFA 

.19 

.23 

.28 

.04 

.01 

-.02 

NRA 

.47 

.57 

.67 

-.07 

-.15 

-.24 

MIA 

.35 

.44 

.53 

.14 

.07 

-.02 

ABA 

.44 

.55 

.65 

.03 

-.08 

-.18 

NAA 

.48 

.58 

.68 

-.11 

-.19 

-.27 

CRI. 

.05 

.01 

-.07 

.14 

.16 

.19 

FI  L 

.11 

.03 

-.09 

.32 

.35 

.38 

NFL 

.07 

-.04 

-.21 

.31 

.34 

.40 

MIL 

.07 

-.09 

-.26 

.21 

.26 

.32 

AB1. 

.34 

.25 

.16 

.35 

.37 

.39 

NIL 

.09 

-.02 

-.17 

.36 

.39 

.43 

Notes.  Test  abbreviations  defined  in 
Applicant  Sample;  YS=  1980  Youth  Sample. 

Figure  1. 

CS  =  Computing  Sample;  AAS  = 

Air  Force 

41 


Table  24.  Validities  of  ASVAB  and  LAMP  Tests  for  Study  2  Accuracy  and  Latency  Criteria  in  the 
Computing  Sample,  the  Air  Force  Applicant  Sample,  and  the  1980  Youth  Sample _ 


ACCURACY 

LATENCY 

Test 

CS 

AAS 

YS 

as 

AAS 

YS 

ASVAB 

GS 

.21 

.42 

.57 

-.19 

-.31 

-.40 

AR 

.34 

.53 

.65 

-.18 

-.29 

-.39 

WK 

.17 

.44 

.59 

-.06 

-.22 

-.33 

PC 

.12 

.39 

.52 

-.07 

-.21 

-.30 

NO 

.15 

.29 

.49 

-.12 

-.19 

-.31 

CS 

.11 

.23 

.42 

-.11 

-.18 

-.29 

AS 

.07 

.28 

.42 

-.02 

-.14 

-.23 

MK 

.38 

.54 

.65 

-.17 

-.26 

-.36 

MC 

.29 

.48 

.57 

-.08 

-.20 

-.29 

El 

.18 

.38 

.55 

-.05 

-.16 

-.29 

LAMP 

GKA 

.17 

.40 

.55 

-.21 

-.32 

-.40 

FRA 

.36 

.43 

.48 

-.15 

-.20 

-.25 

MCA 

.40 

.51 

.60 

-.13 

-.21 

-.30 

NIA 

.23 

.28 

.36 

-.02 

-.05 

-.12 

PIA 

.24 

.25 

.29 

.04 

.02 

-.02 

CRA 

.28 

.31 

.34 

.07 

.04 

-.01 

ABA 

.44 

.58 

.68 

-.07 

-.18 

-.29 

WAA 

.39 

.53 

.62 

-.14 

-.24 

-.3.3 

NAA 

.46 

.56 

.65 

-.14 

-.22 

-.31 

GKL 

-.15 

-.39 

-.55 

.27 

.36 

.43 

NIL 

-.11 

-.22 

-.37 

.37 

.41 

.47 

PIL 

-.08 

-.19 

-.32 

.38 

.41 

.47 

ABL 

.30 

.23 

.15 

.29 

.30 

.32 

CRL 

-.17 

-.25 

-.36 

.28 

.32 

.37 

Notes.  Test  abbreviations  defined  in 

Figure  1. 

CS  =  Computing  Sample;  AAS 

=  Air  Force 

Applicant  Sample;  YS  -  1980  Youth  Sample. 


42 


Table  25.  Contribution  of  LAMP  Tests  to  ASVAB  Tests  in  Accounting  for  Study  1  Criterion  Variance 

in  the  USAF  Applicant  Sample 


R2  Chg. 

F 

P  value 

Annrouch  1 

ACCURACY 

ASVAB  to  LAMP 

.035 

3.319 

.000 

LAMP  to  ASVAB 

.198 

13.412 

.000 

Overall  R  =  .743 

R2-.552  Adj  R2  =  ,525 

LATENCY 

ASVAB  to  LAMP 

.005 

.316 

.977 

IAMPto  ASVAB 

.211 

8.129 

.000 

Overall  R  =  .525 

R2=.2',l  Adj  R2  =  .228 

Approach  2 

ACCURACY 

ASVAB  LAMP  LAMP  +  ASVAB 

L  TO  A 

R 

.595  .719 

.713 

— 

"> 

R~ 

.354  .516 

.552 

.198 

Adj.  R2 

.339  .500 

.525 

.186 

LATENCY 

R 

.245  .515 

.521 

... 

R2 

.060  .265 

.271 

.211 

Adj  R2 

.039  .240 

.228 

.189 

Notes.  L  to  A  reports  the  estimated  contribution  of  LAMP  tests  to  ASVAB  tests. 


Table  26.  Contribution  of  LAMP  Tests  to  ASVAB  Tests  in  Accounting  for  Study  1  Criterion  Variance 

in  the  1980  Youth  Sample 


R2chg. 

F 

P  value 

AcBctadU 

ACCURACY 

ASVAB  to  LAMP 

.029 

3.411 

.(XX) 

LAMP  to  ASVAB 

.160 

12.411 

.000 

Overall  R  =  .798 

R2=.637  Adj  R2 

=  .616 

LATENCY 

ASVAB  to  LAMP 

.005 

.330 

.973 

LAMP  to  ASVAB 

.201 

8.129 

.000 

Overall  R  =  . 551 

R2  =  .304  Adj  R2 

II 

& 

Approach* 

ACCURACY 

ASVAB 

LAMP 

LAMP+ASVAB 

LTO  A 

R 

.619 

.780 

.798 

— 

R2 

.477 

.608 

.637 

.160 

Adj.  R2 

.465 

*94 

.616 

.151 

LATENCY 

R 

.320 

.546 

.551 

... 

R2 

.103 

.298 

.304 

.201 

Adj  R2 

.082 

.274 

.263 

.181 

Notes.  L  to  A  reports  the  estimated  contribution  of  LAMP  tests  to  ASVAB  tests. 
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Table  27.  Contribution  of  LAMP  Tests  to  ASVAB  Tests  in  Accounting  for  Study  2  Criterion  Variance 


Amoash  1 


P  value 


ACCURACY 


ASVAB  to  LAMP 
LAMP  to  ASVAB 

Overall  R  =  . 743  R2=.552 

ASVAB  to  LAMP 
LAMP  to  ASVAB 

Overall  R  =  . 570  R2  =  .325 


.031 

.176 

Adj  R2  =  .526 


LATENCY 

.026 

.192 


Adj  R“  =  .285 


2.831  .002 

11.395  .000 


1.584  .109 

8.260  .000 


Approach  2 


ACCURACY 


ASVAB 

LAMP 

LAMP  +  ASVAB 

LTO  A 

R 

.613 

.722 

.743 

... 

R2 

.376 

.521 

.552 

.176 

Adj.  R2 

.361 

.505 

.526 

.165 

LATENCY 

R 

.364 

.547 

.570 

— 

R2 

.133 

.299 

.325 

.192 

Adj  R2 

.112 

.275 

.285 

.173 

Notes.  L  to  A  reports  the  estimated  contribution  of  LAMP  tests  to  ASVAB  tests. 
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Table  28.  Contribution  of  LAMP  Tests  to  ASVAB  Tests  in  Accounting  for  Study  2  Criterion  Variance 
_ in  the  1980  Youth  Sample _ _ 


R2  Chg. 

F 

P  value 

AfifiDttdLi 

ACCURACY 

ASVAB  to  LAMP 

.027 

3.054 

.002 

LAMP  to  ASVAB 

.139 

11.396 

.000 

Overall  R-.805 

R2  =  .647  Adj  R2  =  .627 

LATENCY 

ASVAB  to  LAMP 

.028 

1.799 

.059 

LAMP  to  ASVAB 

.178 

8.255 

.(XX) 

Overall  R=. 611 

R2=.374  Adj  R2=.337 

Approach  ; 

ACCURACY 

ASVAB  LAMP  LAMP  +  ASVAB 

L  TO  A 

R 

.713  .788 

.805 

— 

R2 

.508  .621 

.647 

.139 

Adj.  R2 

.497  .608 

.627 

.130 

LATENCY 

R 

.442  ,588 

.611 

— 

R2 

.196  .346 

.374 

.178 

Adj  R2 

.176  .324 

.337 

.161 

Notes.  L  to  A  reports  the  estimated  contribution  of  LAMP  tests  to  ASVAB  tests. 
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Tabic  29.  Summary  of  LAMP  Test  Contributions  to  ASVAB  in  Accounting  for  Study  1  and  Study  2 

Accuracy  and  Latency  Criterion  Variance 

Approach  1 


ACCURACY 

LATENCY 

.Study) 

Sludy2 

Studyl 

Sludv2 

Computing  Sample 

23.6% 

22.2% 

22.0% 

20.7% 

A.F.  Applicants 

19.8% 

17.6% 

21.1% 

19.2% 

Youth  Sample 

16.0% 

13.9% 

20.1% 

17.8% 

ADDrouch  2 

ACCURACY 

LATENCY 

Studvl 

Studv2 

Studvl 

Studv2 

Computing  Sample 

22.0% 

20.6% 

19.7% 

18.5% 

A.F.  Applicants 

18.6% 

18.9% 

16.5% 

17.3% 

Youth  Sample 

15.1% 

18.1% 

13.0% 

16.1% 

Notes.  All  values  arc  in  terms  of  the  pereent  valid  variance  added  by  the  LAMP  tests  to 
ASVAB  tests  in  accounting  for  criterion  variance. 

While  the  difference  in  R2s  for  the  full  and  restricted  model  is  the  normal  way  of  defining  the 
unique  contribution  of  one  set  of  tests  to  another  in  accounting  for  criterion  variance,  Brogden  (1946) 
argues  that  variation  in  the  efficiency  of  a  selective  or  predictive  instrument  (either  a  test  or  test 
composite)  is  a  direct  linear  function  of  variation  of  its  product-moment  correlation  (or  multiple 
correlation)  with  the  predicted  variable.  He  demonstrates  that  the  correlation  (or  multiple  correlation) 
coefficient  directly  indicates  the  proportion  of  maximum  saving  that  is  actually  obtained  with  use  of  the 
selection  instrument,  where  the  maximum  saving  is  that  obtained  by  selecting  on  the  criterion.  Data  in 
Table  30  arc  provided  to  relleel  the  predictive  efficiency  of  ASVAB  and  LAMP  tests  in  accordance 
with  this  view,  The  values  in  the  first  two  columns  of  numbers  arc  the  multiple  correlation  coefficients 
for  the  ASVAB  and  LAMP  tests  for  the  various  gates  course  criteria  obtained  in  the  computing 
sample,  in  the  USAF  applicant  sample,  and  in  the  1980  youth  sample.  The  values  in  the  third  column 
are  the  multiple  correlations  for  the  combined  ASVAB  and  LAMP  tests  for  these  same  criteria.  The 
last  column  reports  tiie  differences  in  full  model  multiple  correlation  coefficients  and  the  ASVAB 
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multiple  correlation  coefficients,  which,  according  to  Brogdcn  would  indicate  the  increase  in  predictive 
efficiency  contributed  by  the  LAMP  tests. 

Table  30.  Summary  of  Increases  in  Efficiency  Contributed  by  LAMP  tests  to  ASVAB  Tests  in  the 
Prediction  of  Accuracy  and  Latency  Criteria* 


ACCURACY,  STUDY  1 


Increased 


Predictive  Efficiency 

Predictive 

ASVAB 

LAMP 

A  +  L  Efficiencv 

Computing  Sample 

.478 

.656 

.682  .204 

A.F.  Applicants 

.595 

.719 

.743  .148 

Youth  Sample 

.691 

.780 

.798  .098 

ACCURACY,  STUDY  2 

Computing  Sample 

.460 

.635 

.659 

.199 

A.F.  Applicants 

.613 

.722 

.743 

.130 

Youth  Sample 

.713 

.788 

.805 

.092 

LATENCY,  STUDY  1 

Computing  Sample 

.161 

.490 

.496 

.335 

A.F.  Applicants 

.245 

.515 

.521 

.276 

Youth  Sample 

.320 

.546 

.551 

.231 

LATENCY,  STUDY  2 

Computing  Sample 

.267 

.507 

.526) 

.259 

A.F.  Applicants 

.364 

.547 

.570 

.206 

Youth  Sample 

.442 

.588 

.611 

.169 

Notes.  Predictive  efficiency  is  defined  as  the  expected  value  of  the  gain  effected  by  selecting  with  the 
selection  composite  to  the  gain  that  would  be  effected  by  selecting  on  the  criterion  itself.  See  the  text 
for  addition  comments. 


GENERAL  DISCUSSION 

The  present  paper  is  atypical  of  those  usually  published  in  the  LAMP  scries  in  that  it  reports 
the  results  of  applied  i  ether  than  basic  research.  It  contributes  little  if  anything  to  our  understanding  of 
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how  individuals  think,  remember,  solve  problems,  or  process  information  in  acquiring  new  knowledge 
or  skills.  However,  basic  research  on  cognition  supported  by  the  Air  Force  should  have  some 
probability  of  ultimate  payoff  in  terms  of  improving  selection,  assignment,  training,  or  some  other 
aspect  of  the  personnel  system.  Applied  studies  need  to  be  conducted  on  occasion  to  sec  if  progress  is 
being  made  toward  one  or  more  of  these  goals.  The  levels  of  incremental  validities  found  here  at  the 
very  least  attest  to  the  fact  that  the  LAMP  approach  is  promising.  However  a  number  of  factors  should 
be  considered  in  evaluating  the  final  results,  some  of  which  are  briefly  treated  in  the  following 
paragraphs. 

Nature  of  the  Learning  Task 

The  learning  task  used  in  the  present  study  was  meaningful,  but  it  was  somewhat  unusual  in 
the  sense  that  the  airmen  used  as  subjects  had  little  or  no  incoming  knowledge  specific  to  the  materials 
being  taught.  That  is,  it  represented  a  novel  learning  situation.  This  is  one  of  the  reasons  why  ASVAB 
measures  of  technical  and  verbal  knowledge,  which  have  substantial  validity  for  many  technical  school 
grades,  had  such  low  validities  in  the  present  study.  It  is  anticipated  that  LAMP  tests  will  add 
considerably  less  valid  variance  to  the  ASVAB  tests  in  predicting  technical  school  training  grades  than 
they  did  in  predicting  the  criteria  used  in  the  present  study.  Most  technical  school  grades  reflect 
academic  learning,  and  ASVAB  tests  historically  have  done  well  in  predicting  individual  differences  in 
academic  learning.  However,  it  is  anticipated  that  the  value  of  LAMP  tests  will  be  revealed  against 
criteria  involving  the  development  of  procedural  skills,  which  tends  to  take  place  in  the  job 
environment. 


Concurrency  Effects 

No  concurrency  effects  were  found  in  the  present  study.  This  was  somewhat  of  a  surprise, 
since  one  would  expect  abilities  measured  at  the  time  of  entry  into  a  training  course  to  have  higher 
validities  than  abilities  measured  at  some  prior  point  in  time.  To  the  extent  that  this  expectation  was 
the  case,  the  LAMP  tests  would  have  had  an  advantage  over  the  ASVAB  tests,  which  were 
administered  some  months  earlier.  Some  possible  explanations  for  the  failure  to  find  concurrency 
effects  are  given  below:. 

1)  It’s  simply  a  matter  of  ASVAB  forms.  Several  forms  of  the  ASVAB  were  administered  in 
the  operational  testing  stations,  whereas  a  single  form  was  used  in  the  retesting  session.  Perhaps,  for 
some  unknown  reason,  the  readministration  form  was  inferior  to  the  forms  used  in  the  operational 
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testing  stations.  This  would  be  a  simple  explanation,  but  one  which  is  difficult  to  accept  by  anyone  who 
knows  the  care  taken  in  ASVAB  construction  and  norming. 

2)  The  subjects  were  not  motivated  to  do  their  best  on  the  ASVAB  retest,  but  regained 
motivation  when  faced  with  the  novel  computerized  tests  in  the  LAMP  battery. 

3)  The  ASVAB  retest  scores  were  poorer  measures  of  working  memory  than  were  the 
original  test  scores.  It  is  possible  that  there  was  differential  forgetting  of  procedural  and  declarative 
knowledge  between  test  and  retest.  If  retention  abilities  do  not  predict  the  acquisition  of  a  novel  skill, 
the  addition  of  retention  factors  to  the  concurrent  ASVAB  would  lower  its  validity. 

4)  There  is  little  or  no  concurrency  effect  since  abilities  do  not  change  substantially  during  the 
interval  between  test  and  retest.  This  is  the  third  study  to  suggest  that  currency  effects  arc  small. 
Perhaps  they  are  in  fact  of  little  consequence. 


Common  Methods 

There  is  a  question  of  how  much  of  the  contribution  of  LAMP  tests  to  ASVAB  tests  can  be 
attributed  to  the  use  of  common  methods.  It  is  true  that  the  LAMP  tests  and  the  logic  gates  courses 
were  both  administered  by  microcomputer,  while  the  ASVAB  tests  were  administered  by  paper  and 
pencil. 


It  would  be  difficult  to  accept  that  commo  methods  played  a  significant  role  in  determining 
the  relationships  of  LAMP  tests  with  accuracy  criteria.  If  an  individual  correctly  analyzed  the  input 
signals  to  a  logic  gate  and  knew  the  output  to  be  either  high  or  low,  it  should  not  have  made  much 
difference  whether  the  answer  was  recorded  by  marking  an  answer  sheet  or  pressing  a  computer  key. 

The  same  cannot  be  said  with  respect  to  predicting  latency  criteria.  In  the  computing  sample, 
none  of  the  ASVAB  test  scores  were  significantly  related  to  latency  scores  from  the  logic  gates  courses, 
while  all  of  the  latency  scores  from  the  LAMP  tests  showed  validity  for  the  course  latency  data.  A 
basic  question  is  "How  much  of  an  unfair  advantage  did  the  use  of  common  methods  give  to  the  LAMP 
tests?".. 


To  answer  this  question,  one  first  should  decompose  the  criterion  variance  into  two  parts:  1) 
that  part  which  was  relevant  to  the  skill  being  taught,  and  2)  that  part  which  was  not  relevant  to  the  skill 
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being  taught.  The  relevant  variance  in  the  logic  gates  course  latency  data  was  associated  with  the  time 
required  for  subjects  to  mentally  process  information  to  determine  gate  output  signals.  The  non- 
relcvant  variance  was  associated  with  the  time  required  to  record  answers  using  the  computer 
keyboard.  One  might  argue,  then,  that  the  variance  associated  with  common  methods  may  have  given 
the  LAMP  tests  some  unfair  advantage  with  respect  to  answer  entry -but  not  with  respect  to 
measuring  the  time  required  for  mental  processing.6 


Processing  Accuracy 

The  LAMP  measures  of  processing  accuracy  made  a  small  but  significant  contribution  to  other 
LAMP  measures  in  predicting  accuracy  criteria  from  the  gates  courses.  These  scores  were  extracted 
from  simple  processing  speed  tests  having  high  accuracy  levels.  Within  testing  sessions,  individuals  who 
made  errors  on  one  of  these  tests  tended  to  be  the  same  individuals  who  made  errors  on  the  others. 
However,  it  is  important  to  demonstrate  that  such  tendencies  arc  stable  across  time.  If  they  arc  not, 
then  the  present  analyses  may  have  capitalized  on  some  common-factor  variance  between  the  LAMP 
tests  and  the  logic  gate  course  criteria  which  will  not  hold  up  in  a  predictive  sense. 


Testing  Time 

Testing  time  is  important  when  one  is  considering  the  utility  of  new  tests  for  possible  inclusion 
in  future  operational  batteries.  It  is  reasonable  to  ask  whether  the  apparent  advantage  of  LAMP  tests 
in  the  present  study  was  at  the  expense  of  excessive  testing  time.  Unfortunately,  it  is  not  possible  to 
give  precise  time  requirements  for  the  LAMP  tests  or  for  the  logic  gates  courses,  since  the  subjects 
proceeded  at  their  own  rate.  However,  the  question  can  be  answered  in  a  gross  fashion.  About  one 
hour  was  required  for  the  slowest  individuals  to  complete  the  course  in  Study  1,  and  about  one-and- 
one  half  hours  was  required  for  the  slowest  individuals  to  complete  the  course  in  Study  2.  Since  only 
three  hours  of  testing  time  was  available,  a  gross  estimate  is  that  the  LAMP  tests  took  about  two  hours 
for  administration  in  Study  1  and  one-and-one-half  hours  for  administration  in  Study  2.  In  each 
instance,  this  is  less  time  than  was  required  for  administration  of  the  ASVAB. 


6()ne  of  the  weaknesses  of  paper-and-pencil  tests  is  that  they  do  not  provide  a  means  for  accurately 
measuring  the  speed  of  mental  processing.  In  the  case  of  solving  logic  gate  problems,  precision  is 
needed  to  a  tenth  of  a  second  in  order  to  reliably  discriminate  between  subjects.  The  wave  of  the 
future  is  computer-administered  courses  and  computerized  job  tasks.  The  day  may  come,  when  even 
the  time  associated  with  keyboard  entry  is  course  and/or  job  related. 
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Of  course,  it  should  be  recognized  that  neither  the  ASVAB  nor  the  LAMP  tests  were  designed 
with  the  logic  gates  course  in  mind.  Both  test  sets  were  designed  to  measure  knowledge  and  abilities 
relevant  to  learning  in  general.  It  is  interesting  to  note  that  one  of  the  LAMP  tests  with  the  highest 
validity  for  the  accuracy  criteria  was  Grammatical  Reasoning,  which  took  only  five  minutes  for 
administration. 

Other  Considerations 

Two  factors  stand  out  as  explaining  individual  differences  in  the  acquisition  of  declarative 
knowledge  about  gate  rules  and  the  development  of  procedural  skill  in  solving  logic  gate  problems. 
Information  processing  speed  and  working  memory  capacity  were  the  best  predictors  of  procedural 
skill  development,  and  working  memory  capacity  alone  was  by  far  the  best  predictor  of  rule  learning. 

The  fact  that  ASVAB  does  not  contain  good  measures  of  processing  speed  accounts  for  its 
inability  to  predict  procedural  skill  development  in  the  present  study.  The  ASVAB  docs  contain  three 
tests  which  loaded  on  the  Working  Memory  factor,  and  they  accounted  for  essentially  all  of  the 
ASVAB  validity  for  gates  course  accuracy  criteria.  However,  the  ASVAB  measures  of  working 
memory  are  heavily  dependent  on  declarative  and  procedural  knowledge  gained  in  academic  schooling, 
such  as  taking  a  square  root  or  setting  up  equations  to  solve  for  an  unknown.  On  the  other  hand,  the 
LAMP  tests  of  working  memory  yielded  higher  validities  for  the  gates  course  accuracy  criteria,  and  are 
less  dependent  on  academic  knowledge.  For  example,  in  the  computing  sample ,  the  Arithmetic 
Reasoning  and  Mathematics  Knowledge  test  validities  for  the  average  accuracy  scores  across  all  blocks 
in  Study  2  were  .34  and  .38,  respectively,  while  the  Grammatical  Reasoning,  and  Numerical 
Assignment  tests  in  the  LAMP  battery  correlated  .44,  and  .46  with  this  same  criterion.  This  advantage 
was  maintained  even  in  the  data  corrected  to  the  USAF  Applicant  sample,  where  the  values  were  .53 
and  .54  for  the  two  ASVAB  tests,  compared  with  .58  and  .56  for  the  two  LAMP  tests. 

An  important  side  benefit  of  using  processing-based  predictors  is  that  they  arc  not  only  more 
valid  against  certain  criteria,  but  they  also  tend  to  be  less  related  to  Race  and  Sex.  For  example,  in  the 
Study  2  computing  sample,  the  Arithmetic  Reasoning  test  correlated  .22  with  Race  (1/0=  Non- 
Black/Black)  while  the  Mechanical  Comprehension  test  correlated  .33  with  Race  and  .29  with  Sex 
(l/0  =  Male/Female).  In  contrast,  the  Grammatical  Reasoning  test  (the  highly  valid  5-minute  lest), 
correlated  only  .11  with  Race  and  -.01  with  Sex.  There  appears  to  be  some  merit  in  considering  some 
type  of  processing-based  working  memory  test  for  future  forms  of  the  ASVAB. 
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CONCLUSIONS 


Tabic  29  presents  a  summary  of  the  major  findings  from  Studies  1  and  2.  In  general,  the 
LAMP  tests  added  about  20%  unique  valid  variance  to  the  ASVAB  in  predicting  both  accuracy  and 
latency  block  averages  in  the  two  logic  gate  courses.  When  the  data  were  corrected  for  restrictions  in 
range  to  the  USAF  applicant  sample,  the  contributions  of  LAMP  tests  to  ASVAB  tests  still  ranged 
from  17%  -  20%  .  The  data  in  Table  30  show  substantial  contributions  of  the  LAMP  tests  to  the 
ASVAB  tests  in  terms  of  increased  predictive  effcciency,  where  predictive  efficiency  is  defined  as  the 
expected  value  of  the  gain  effected  by  selecting  with  the  selection  composite  to  the  gain  that  would  be 
effected  by  selecting  on  the  criterion  itself. 

Analyses  indicated  that  the  advantage  of  the  LAMP  tests  over  the  ASVAB  tests  was  not  the 
result  of  a  concurrency  effect.  Nor  was  it  obtained  at  the  expense  of  excessive  testing  time.  Although 
the  LAMP  tests  and  criterion  tests  used  common  methods,  arguments  were  presented  contending  that 
most  of  the  covariance  associated  with  common  methods  was  meaningful  to  the  task  being  learned  and 
performed. 

While  the  findings  in  the  current  study  were  for  learning  and  skill  development  in  a  single 
subject-matter  area,  this  subject-matter  was  selected  because  of  its  relevance  to  meaningful  USAF 
functions.  The  results  arc  viewed  as  being  highly  encouraging  with  respect  to  the  potential  contribution 
of  the  Learning  Abilities  Measurement  Program  (LAMP)  to  the  USAF  personnel  system. 
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APPENDIX  A 

DESCRIPTIONS  OF  ASVAB  AND  LAMP  TESTS 


The  first  ten  tests  listed  are  from  the  ASVAB  and  their  descriptions  arc  based  on  information 
in  the  Department  of  Defense  Bulletin  DOD  1340.12AA  (1984). 

ASVAB  Tests  Used  in  Study  I 

General  Science.  This  test  contains  35  standard  vocabulary  items,  such  as  "Which  of  the 
following  foods  contain  the  most  iron?  (a)  eggs,  (b)  liver,  (c)  candy,  (d)  cucumber." 

Arithmetic  Reasoning.  This  test  consists  of  30  arithmetic  word  problems,  such  as  the 
following:  "Pat  put  in  a  total  of  16  1/2  hours  on  a  job  during  5  days  of  the  past  week.  How  long  is  Pat’s 
average  workday?  (a)  3  hours,  (b)  3  hours,  15  minutes,  (c)  3  hours,  18  minutes,  (d)  3  hours,  25 
minutes." 

Word  Knowledge.  This  test  consists  of  35  standard  vocabulary  items,  such  as  'The  wind  is 
variable  today,  (a)  mild,  (b)  steady,  (c)  shifting,  or  (d)  chilling." 

Paragraph  Comprehension.  This  test  consists  of  15  paragraphs,  each  l-to-3  sentences  long, 
followed  by  a  multiple-choice  question  of  the  paragraph’s  content.  An  example  is  the  following: 

From  a  building  designer’s  standpoint,  three  things  that  make  a  home  livable  are  the  client,  the 
building  site,  and  the  amount  of  money  the  client  has  to  spend.  According  to  the  passage,  to  make  a 
home  livable 

(a)  it  can  be  built  on  any  piece  of  land; 

(b)  the  design  must  fit  the  designer’s  income; 

(c)  the  design  must  fit  the  owner’s  income  and  site; 

(d)  the  prospective  piece  of  land  makes  little  difference. 

Numerical  Operations.  This  is  a  speeded  test,  consisting  of  50  number-fact  items  (e.g.,  2x6 
-=  ?  (a)  4,  (b)  8,  (c)  3,  (d)  12).  Examinees  arc  given  10  minutes  to  finish  as  many  of  the  items  as  they 
can. 
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Coding  Speed,  This  is  a  speeded  test  consisting  of  84  items  designed  to  measure  how  quickly 
an  examinee  can  find  a  number  in  a  table.  An  item  consists  of  a  word  followed  by  five  4-digit  number 
strings  (e.g.,  (a)  6456,  (b)  7150,  (c)  8930,  (d)  9645).  The  subject’s  task  is  to  (a)  look  up  the  word’s 
number  code  in  a  key  consisting  of  10  word-code  pairs  placed  on  top  of  the  page,  then  (b)  select  the 
letter  associated  with  that  number  code.  Subjects  arc  given  10  minutes  to  complete  as  many  items  as 
they  can. 

Auto  and  Shop  Information.  This  tests  consists  of  25  questions  about  automobiles,  shop 
practices,  and  the  use  of  tools.  An  example  question  is:  "A  chisel  is  used  for  (a)  prying,  (b)  cutting,  (e) 
twisting,  (d)  grinding."  Some  of  the  questions  include  drawings  of  common  tools  or  parts. 

Mathematics  Knowledge.  This  test  consists  of  25  arithmetic  word  problems  (primarily 
algebra,  but  also  area,  square  root,  percentage,  and  simple  geometry),  such  as  the  following:  "If  3X  = 
5,  then  X  =  :  (a)  -2,  (b)  -5/3,  (c)  3/5,  (d)  -3/5." 

Mechanical  Comprehension.  This  test  consists  of  25  questions  relating  to  general  mechanical 
and  physical  principles.  The  questions  are  normally  accompanied  by  drawings.  For  example,  a  drawing 
may  include  three  meshed  gears  with  an  arrow  indicating  the  direction  of  rotation  of  one  of  the  gears. 
The  question  might  be  "Which  of  the  other  gears  is  moving  in  the  same  direction  as  gear  1?  (a)  Gear 
2,  (b)  Gear  3,  (c)  Both  of  the  other  gears,  (d)  Neither  of  the  other  gears." 

Electrical  Information.  This  test  consists  of  20  questions  relating  to  knowledge  of  electrical, 
radio,  and  electronics  information.  Some  of  the  questions  contain  drawings,  while  others  do  not.  The 
following  is  an  example  question:  "Which  of  the  following  has  the  least  resistance? 

(a)  Iron,  (b)  Wood,  (c)  Silver,  (d)  Rubber." 


LAMP  Tests  Included  in  Study  l 

Choice  Reaction  Time.  This  test  consists  of  5  blocks  of  20  items,  for  a  total  of  1(H).  In  each 
item,  a  block  of  4  asterisks  is  presented  on  the  right  center  or  left  center  of  the  screen.  If  the  asterisks 
appear  on  the  right  center,  the  subject  is  requested  to  press  the  ‘L’  key.  If  they  are  on  the  lelt  center, 
the  subject  is  requested  to  press  the  ‘D’  key.  Latency  in  milliseconds  and  ‘CORRECT  or  'WRONG' 
feedback  is  provided  after  each  response.  There  is  a  1-sccond  intertrial  time  between  items.  Summary 
feedback  is  provided  after  each  of  the  five  blocks.  Although  individuals  are  requested  to  respond  as 
quickly  as  possible  without  making  mistakes,  mistakes  are  in  fact  made  by  most  subjects.  The  primary 
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score  extracted  from  the  response  data  is  ‘median  latency’  in  milliseconds.  A  'percent  correct’  score  is 
also  computed  and  is  used  in  the  analyses. 

Physical  Identity.  This  test  consists  of  5  blocks  of  20  items  each,  for  a  total  of  100  items.  In 
each  item,  two  words  arc  presented  on  the  screen,  one  above  the  other.  The  words  arc  either  both  in 
upper-ease  letters  or  both  in  lower-case  letters.  In  50  percent  of  the  trials,  the  two  words  are  physically 
identical.  In  the  other  50  percent  of  the  trials,  the  words  are  different,  even  though  the  case  for  the  two 
words  is  the  same.  There  is  a  warning  asterisk  prior  to  displaying  each  item.  The  subject  is  requested 
to  press  the  ‘L’  key  if  the  two  words  arc  physically  identical,  or  the'D’  key  if  they  are  not.  Response 
latency  feedback  is  provided  after  each  correct  response,  and  the  word  ‘WRONG’  is  presented,  along 
with  the  sound  of  a  buzzer,  after  each  incorrect  response.  Summary  feedback  is  provided  at  the  end  of 
each  block.  As  in  the  ease  of  the  Choice  Reaction  Time  test,  both  median  latency  and  accuracy  scores 
arc  obtained  from  the  test  results. 

Name  Identity.  As  in  the  ease  of  the  Choice  Reaction  Time  and  Physical  Identity  tests,  this 
lest  contains  5  blocks  of  20  items  each,  for  a  total  of  100  items.  Two  words  are  presented  in  the  middle 
of  the  screen,  one  above  the  other.  One  of  the  words  is  presented  in  upper-ease  letters,  while  the  other 
is  presented  in  lower-case  letters.  In  50  percent  of  the  items,  the  two  words  arc  the  same;  in  the  other 
50  percent,  the  two  words  are  different  (have  a  different  name).  The  subject  is  requested  to  press  the 
‘L’  key  if  the  two  words  have  the  same  name,  or  the  ‘D\  key  if  they  have  different  names.  The  display 
format  and  the  feedback  provided  is  the  same  as  in  the  Physical  Identity  test,  above.  Both  latency  and 
accuracy  scores  are  derived  from  the  test. 

Meaning  Identity.  Meaning  Identity  is  another  information  processing  speed  measure  which 
contains  5  blocks  of  20  items  each.  In  each  item,  two  words  are  displayed,  one  above  the  other,  in  the 
center  of  the  screen  immediately  after  a  warning  asterisk.  Both  words  are  presented  in  lower-case 
letters.  The  subject  is  instructed  to  press  the  ‘L’  key  if  the  words  have  a  similar  meaning  (are 
synonyms),  or  the  ‘D’  key  if  they  do  not  have  similar  a  similar  meaning.  The  words  were  selected  to 
minimize  the  role  of  knowledge,  since  the  goal  is  to  measure  speed  of  memory  search,  memory 
retrieval,  and  feature  comparison.  As  in  the  case  of  the  other  processing  speed  measures,  latency 
feedback  is  given  on  correct  responses  and  ‘WRONG’  feedback  is  given  on  incorrect  responses.  Both 
median  latency  and  percent  correct  scores  were  computed  for  all  subjects. 

Number  Fact  Retrieval.  Subjects  arc  given  four  sets  of  simple  arithmetic  problems:  50 
addition  (4  +■  2),  50  multiplication  (3  X  2),  25  subtraction  (8  -  4),  and  25  division  (9  /  3).  In  each 
instance,  the  problem  is  displayed  in  the  middle  of  the  screen,  with  a  proposed  answer  centered  below 
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it.  The  subject  is  instructed  to  press  the  ‘L’  key  if  the  proposed  answer  is  correct,  or  the  ‘D’  key  if  the 
proposed  answer  is  incorrect.  Each  problem  is  preceded  with  a  warning  star.  Median  latency  and 
percent  correct  scores  are  obtained  for  each  subject. 

AB  Grammatical  Reasoning.  AB  Grammatical  Reasoning  is  a  computerized  version  of 
Baddele/s  (1968)  "3  minute  test  of  grammatical  reasoning."  Subjects  determine  whether  a  sentence 
such  as  "A  follows  B"  is  consistent  with  an  arrangement  of  the  letters  A  and  B  (either  "AB"  or  "BA"). 
The  sentence  and  the  letters  appear  on  the  screen  simultaneously.  Sentences  arc  varied  in  relation 
(precedes  vs.  follows),  voice  (active  vs  passive),  negation  (present  vs.  passive),  subject  (A  vs.  B),  and 
truth  (true  vs.  false).  There  arc  2"*  =  32  X  2  (replications)  =  64  items,  divided  into  four  blocks.  The 
computer  provides  latency  and  ‘CORRECT’  or  ‘WRONG’  feedback  after  each  item,  and  summary 
information  at  the  end  of  each  block.  The  computer  displays  items  one-at-a-time,  with  no  time  limit. 
Both  median  latency  and  percent  correct  scores  arc  computed  for  each  subject. 

123  Number  Reduction.  The  123  Number  Reduction  test  is  a  modification  of  Thurstonc  and 
Thurstone’s  (1941)  ABC  Test.  Subjects  arc  asked  to  translate  2-,  3-,  4-  ,5-  and  6-digit  numbers 
according  to  two  rules.  Rule  1  is  to  rewrite  two  successive  same  digits  (c.g.,  2  2)  to  that  digit  (i  e.,  2). 
Rule  2  is  to  rewrite  two  successive  different  digits  (c.g.,  3  2)  to  the  third  digit  (i.e.,  1).  The  subject  is 
instructed  to  evaluate  pairs  left-to-right  and  to  enter  the  final  digit  resulting  from  the  complete 
recoding  sequence.  This  test  was  designed  to  measure  working  memory,  since  it  involves  processing  in 
the  presence  of  a  memory  load.  The  test  is  scored  for  percent  correct  only. 

Digit  Span.  Digit  Span  is  another  test  designed  to  measure  working  memory.  The  computer 
displays  5,  7,  or  9  digits  successively  for  1  second  each,  left-to-right,  2  characters  apart  on  the  display 
screen.  Immediately  after  the  presentation,  probe  digits  are  presented  one  at-a-time  from  either  the 
beginning  or  end  of  the  list.  The  subject’s  task  is  to  indicate  (by  pressing  the  ‘L’  or  ‘D’  key)  whether 
the  probe  matches  the  displayed  digit.  After  the  subject  responds,  a  probe  for  the  next-to-Iast  digit  is 
presented  and  subjects  respond  in  the  same  fashion.  On  half  of  the  items,  probes  arc  from  the  end  of 
the  list,  and  on  the  other  half,  from  the  beginning  of  the  list.  From  2  to  5  probes  are  displayed  for  each 
list.  Subjects  arc  given  credit  for  each  of  the  probes  they  correctly  discriminate.  Items  are  blocked  by 
digit  strings  and  arc  displayed  in  order  (5-digit  items  first,  followed  by  7-digit  and  then  two  blocks  of  9 
digit  items).  There  are  eight  5-digit  items  (21  responses),  ten  7-digit  items  (34  responses),  5,  items  in 
the  first  9-digit  block  (21  responses),  and  5  items  in  the  second  9-digit  block  (23  responses),  for  a  total 
of  4  blocks.  The  test  provides  summary  percent  correct  feedback  at  the  end  of  each  block.  It  is  scored 
only  for  percent  correct. 
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ABC  Numerical  Assignment.  This  task,  which  in  previous  studies  has  been  found  to  be  a 
strong  measure  of  working  memory,  consists  of  21  items,  each  requiring  three  responses  (63  probes, 
total),  presented  in  a  single  block.  Each  item  presents  a  sequence  of  screens  in  which  each  of  the 
letters  A,  B,  and  C  arc  assigned  values  or  equations.  A  typical  (but  difficult)  item  is:  (Screen  1)  "A  =  B 
/2";  (Screen  2)  "B  =  C  -  4)";  (Screen  3)  "C  =  8";  (Screen  4)  "B  =  ?";  (Screen  5)  "A  =  ?";  (Screen  6) 
“C  =  ?".  Letters  were  assigned  cither  values  (e.g.,  A  =  5)  or  equations  (e.g.,  A  =  B/2).  In  equation 
assignments,  it  was  sometimes  possible  to  solve  for  the  equation  with  information  already  known.  In 
other  cases  (such  as  in  the  example  item),  it  is  necessary  to  hold  the  equation  in  memory  and  wait  for 
later  screens  to  provide  missing  information  that  make  it  possible  to  solve  the  equation.  The  test 
allows  unlimited  per-scrcen  study  time,  but  subjects  are  not  able  to  scroll  backwards  to  review  screens. 
The  computer  docs  not  provide  feedback. 


ASVAB  Tests  in  Study  2 

Study  2  involved  use  of  the  same  10  ASVAB  tests  as  described  for  Study  1,  above. 


LAMP  Tests  in  Study  2 

Choice  Reaction  Time.  This  is  the  same  test  as  was  used  in  Study  1,  above. 

Physical  Identity.  This  is  the  same  test  as  was  used  in  Study  1,  above. 

Name  Identity.  This  is  the  same  test  as  was  used  in  Study  1,  above. 

General  Knowledge  Test.  Subjects  are  required  to  enter  the  first  two  letters  of  the  single¬ 
word  answers  to  24  general  knowledge  questions  such  as  “What  is  the  name  of  the  molten  rock  that 
flows  down  the  sides  of  an  active  volcano?"  (LAVA).  The  questions  were  selected  from  an  earlier  100- 
item  form  using  item-analysis  procedures.  The  test  is  scored  for  both  percent  correct  and  latency. 

Word  Association  Test.  The  subject  is  asked  to  associate  9  pairs  of  words  in  each  of  four  study 
blocks.  The  word  pairs  in  a  block  are  presented  one  at-a-time  at  a  rate  of  4  seconds  per  pair..  Each 
study  block  is  followed  by  9-alternative  multiple-choice  items  in  which  the  subject  is  presented  with 
one  member  word  of  a  pair  and  asked  to  identify  the  other  member  word  with  which  it  was  associated. 
The  distractors  for  each  item  arc  selected  from  among  the  18  words  used  in  the  immediate  block.  The 
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subject  is  provided  with  latency  and  accuracy  feedback  for  each  item,  and  summary  accuracy  and 
htcncy  data  at  the  end  of  each  block. 

Figure  Recognition  Test.  In  this  test,  figures  are  drawn  by  connecting  lines  between  points  in 
an  invisible  9  by  9  dot  matrix.  Of  the  over  one-million  potential  figures,  K>0  were  selected  which  were 
judged  to  be  at  least  moderately  encodable.  In  each  of  four  blocks,  20  of  the  figures  were  displayed  in 
a  5  by  4  matrix  for  a  75-sccond  study  time.  The  subjects  arc  asked  to  study  each  figure  so  that  they  will 
recognize  it  if  they  sec  it  again.  Subjects  are  encouraged  to  give  meaning  to  figures  by  associating 
them  with  known  objects  or  scenes.  At  the  end  of  the  75-second  study  period,  40  figures  are  presented 
one-at-a-time.  If  a  figure  is  judged  to  be  one  of  the  20-figure  study  set,  the  subject  is  asked  to  press  the 
‘L’  key.  Otherwise,  the  subject  is  asked  to  press  the  ‘D’  key.  Accuracy  and  latency  feedback  is 
provided  after  each  response,  and  summary  feedback  is  provided  at  the  end  of  each  block.  This  test  is 
scored  for  percent  correct  only. 

Mental  Character  Generation.  In  this  test,  the  subjects  are  asked  to  commit  to  memory  the 
numbers  from  1  to  9  in  the  relative  locations  shown  below: 

1  4  7 

2  5  8 

3  6  9 

For  a  particular  item,  subjects  arc  asked  to  mentally  connect  numbers  with  lines  in  accordance 
with  instructions  displayed  on  successive  screens.  While  processing  a  given  screen,  the  subjects  must 
retain  a  mental  image  of  the  figure  drawn  by  previous  instructions,  and  add  to  this  figure  the  line  called 
for  by  the  present  instruction.  If  the  instructions  are  properly  executed,  at  the  end  of  the  last 
instruction  in  a  set,  the  mental  image  of  a  letter  or  number  will  have  been  generated.  The  subjects  arc 
asked  to  record  this  letter  or  number  by  pressing  the  corresponding  key.  For  example,  here  arc  the 
instructions  for  one  of  the  items: 

screen  1:  connect  3  to  1 
screen  2:  connect  7  to  6 
screen  3:  connect  1  to  6 
screen  4:  ? 

The  correct  answer  to  the  above  item  is  the  letter  ‘V’.  There  are  only  13  items  in  this  test. 
Two  of  them  involve  connecting  2  lines,  two,  3  lines;  four,  4  lines;  three,  5  lines;  and  two,  6  lines.  The 


subjects  arc  permitted  to  move  at  their  own  pace,  and  feedback  is  provided  on  accuracy.  This  test  was 
designed  to  be  a  measure  of  working  memory  involving  spatial  materials.  It  requires  storing  a  mental 
image  while  processing  additional  visual  information. 


Tests  used  in  Study  3 

Study  3  did  not  involve  administration  of  any  tests  from  the  LAMP  program.  It  involved  only 
the  administration  of  three  alternate  forms  of  tests  from  the  ASVAB  (Arithmetic  Reasoning, 
Mathematics  Knowledge,  and  Mechanical  Comprehension).  These  tests  were  selected  as  the  three 
ASVAB  tests  that  were  most  predictive  of  learning  in  the  Logic  Gates  Course.  The  purpose  of  Study  3 
was  to  evaluate  the  possible  influence  of  concurrency  on  computed  validity  coefficients. 
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APPENDIX  B 


FACTOR  ANALYSIS  OF  STUDY  1  AND  STUDY  2  TEST  SCORES 


Study  1  Factor  Matrix 


Tabic  Bl. 

Results  from  Factor  Analysis  of  Study  1  Predictor  Tests 

TEST 

PROCSPD 

WKMEM 

PROCACC 

VERBAL 

TECHKNOW 

NUM/PS 

NIL 

.89 

-.04 

.05 

-.01 

.00 

-.17 

PIL 

.81 

-.07 

.17 

.04 

.04 

-.12 

MIL 

.69 

-.10 

-.03 

-.15 

-.02 

-.08 

CRL 

.50 

-.05 

.08 

.01 

-.03 

-.02 

NAA 

-.09 

.74 

.21 

.11 

.08 

.16 

DSA 

-.20 

.58 

.19 

-.09 

.07 

.03 

MK 

.03 

.58 

-.02 

.30 

.14 

.36 

AR 

-.02 

.55 

.07 

.20 

.27 

.29 

NRA 

-.15 

.54 

.30 

.21 

.08 

.07 

ABA 

.02 

.47 

.30 

.34 

.04 

-.08 

PIA 

.15 

01 

.77 

-.03 

-.00 

.07 

NIA 

p 

• 

.05 

.73 

.04 

.03 

.10 

MIA 

.06 

.20 

.68 

.15 

-.03 

-.04 

CRA 

.07 

.13 

.43 

-.04 

.11 

.05 

NFA 

,03 

.22 

.33 

-.03 

.00 

-.01 

ABL 

.23 

.21 

.32 

-.13 

.03 

-.21 

WK 

-.06 

.24 

-.04 

.76 

.15 

-.13 

GS 

.04 

.11 

-.13 

.63 

.37 

-.11 

PC 

-.07 

.04 

.08 

.61 

.15 

.11 

El 

.05 

.14 

.04 

.31 

.71 

-.07 

AS 

.01 

.02 

.09 

.09 

.69 

-.17 

MC 

-.02 

.32 

.03 

.25 

.60 

-.08 

NO 

-.09 

.17 

.07 

-.06 

-.18 

.67 

NFL 

.50 

oo 

p 

* 

.15 

.06 

.10 

-.56 

CS 

-.20 

.13 

.14 

-.03 

-.11 

.50 

Notes.  Sec  Figure  1  for  test  titles.  Loadings  arc  from  Varimax  rotations  of  Principal  Axis  Factors. 
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Study  1  Factor  Descriptions 


FACTOR  I-  PROCESSING  SPEED 

This  factor  was  defined  solely  by  latency  scores  from  LAMP  tests  designed  to  measure 
processing  speed,  including  the  Name  Identity,  Physical  Identity,  Meaning  Identity,  Choice  Reaction, 
and  Number  Fact  Recall  tests.  No  test  from  the  ASVAB  had  significant  loadings  on  this  factor. 

FACTOR  II-  WORKING  MEMORY 

All  of  the  LAMP  tests  selected  to  be  measures  of  working  memory  had  significant  loadings  on 
this  factor.  The  test  with  the  highest  loading  was  the  Number  Assignment  test  (sometimes  called  the 
ABC  Recall  Test),  which  has  been  the  primary  definer  of  working  memory  in  several  previous  studies 
(Christa!.  1988;  Kyllonen  &  Christal,  1989,  1990).  The  three  ASVAB  tests  with  significant  loadings  on 
this  factor  were  Mathematics  Knowledge,  Arithmetic  Reasoning,  and  Mechanical  Comprehension,  all 
of  which  have  had  significant  loadings  on  the  working  memory  factor  in  previous  analyses  (Christal, 
1988).  In  the  Study  1  factor  analysis,  it  so  happened  than  many  of  the  tests  defining  the  working 
memory  factor  were  based  tin  processing  numerical  content.  However,  the  fundamental  nature  of  the 
factor  is  believed  to  be  the  requirement  for  information  processing  in  the  presence  of  a  memory  load, 
and  unpublished  LAMP  studies  suggest  that  this  factor  may  not  be  heavily  influenced  by  the  type  of 
content  processed.  All  of  the  tests  in  Study  l  required  the  subjects  to  store  intermediate  results  while 
completing  the  processing  demanded  for  problem  solution. 

FACTOR  HI  -  PROCESSINC,  ACCURACY 

This  factor  was  defined  by  accuracy  scores  from  simple  LAMP  processing  speed  measures. 
Most  subjects  made  near-perfect  scores  on  such  tests.  For  example  the  tests  with  the  top  four  loadings 
on  this  factor  were  Physical  Identity,  Name  Identity,  Meaning  Identity,  and  Choice  Reaction,  for  which 
the  mean  percent  accuracy  scores  were  95.5,  95.4,  93.2,  and  98.5,  respectively.  Although  few  errors 
were  made  on  these  simple  processing  tasks,  the  subjects  who  made  the  most  errors  on  one  of  these 
tests  tended  to  be  those  who  also  made  the  most  errors  on  the  others. 
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FACTOR  IV  -  VERBAL 


This  is  the  verbal  factor  which  is  routinely  identified  in  analyses  of  ASVAB  tests.  In  the 
present  study  it  was  defined  by  the  Word  Knowledge,  Paragraph  Comprehension,  and  General  Science 
tests.  None  of  the  LAMP  tests  were  designed  to  measure  verbal  knowledge,  and  none  had  large 
loadings  on  this  factor. 


FACTOR  V  -  TECHNICAL  KNOWLEDGE 

This  is  the  technical  knowledge  factor  which  is  routinely  found  in  factor  analyses  of  the 
ASVAB  tests.  It  was  defined  by  Electrical  Information,  Auto  and  Shop  Information,  Mechanical 
Comprehension,  and  General  Science.  The  only  LAMP  test  that  had  a  significant  loading  on  this 
factor  was  Grammatical  Reasoning,  which  had  a  loading  of  only  .34. 


FACTOR  VI  -  NUMERICAL  FACILITY/PERCEPTUAL  SPEED 

Factor  analyses  of  tests  from  the  ASVAB  normally  yield  a  doublet  factor  defined  by  Numerical 
Operations  and  Coding  Speed.  This  is  often  labeled  as  being  a  perceptual  speed  factor.  In  the  present 
analysis,  the  LAMP  Number  Fact  Recall  test  joined  the  two  ASVAB  tests  in  defining  the  factor,  and 
had  the  second  highest  loading  on  it.  Since  two  of  the  three  tests  defining  the  factor  involve  simple 
numerical  computations  under  speeded  conditions,  it  was  labeled  in  this  study  as  being  a  '  Numerical 
Facility/Pcrceptual  Speed"  factor. 
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Tabic  B2.  Results  from  Factor  Analysis  of  Study  2  Predictor  Tests 


TEST 

VERBAL 

WKMEM 

PROCSPD 

PROCACC 

TECHKNOW 

NUM/PS 

WK 

.73 

.09 

-.04 

.01 

.09 

-.07 

(iKA 

.68 

.21 

-.06 

-.02 

.20 

-.11 

CIS 

.64 

.21 

-.06 

-.01 

.38 

-.00 

GKL 

-.59 

-.10 

.21 

.11 

-.08 

-.35 

PC 

.53 

.05 

-.06 

.04 

.05 

.13 

NAA 

.07 

.71 

-.04 

.22 

.00 

.17 

MK 

.19 

.59 

-.01 

-.03 

.02 

.43 

AR 

.11 

.54 

.06 

-.04 

.21 

.47 

MCA 

.14 

.51 

-.10 

.12 

.13 

.02 

FRA 

.06 

.50 

-.24 

.23 

.09 

-.18 

WAA 

.30 

.46 

-.23 

.20 

.06 

-.11 

ABA 

.36 

.43 

-.00 

.25 

.08 

.12 

PIL 

-.06 

-.07 

.84 

.18 

-.03 

-.19 

NIL 

-.16 

-.03 

.84 

.13 

.09 

-.22 

CRL 

-.08 

-.17 

.51 

.17 

-.01 

-.05 

P1A 

-.00 

.07 

.12 

.76 

-.01 

.14 

NIA 

.07 

.10 

.08 

.66 

-.00 

.08 

CRA 

-.03 

.18 

.11 

.46 

.12 

-.00 

ABL 

-.05 

.26 

.20 

.42 

-.08 

-.19 

AS 

.11 

-.05 

.06 

-.01 

.78 

-.12 

MC 

.19 

.32 

-.02 

-.02 

.63 

-.03 

El 

.29 

.10 

-.01 

.13 

.63 

-.06 

NO 

.02 

.07 

-.18 

.07 

-.10 

.63 

CS 

.00 

.04 

-.19 

.08 

-.15 

.60 

Notch.  See  Figure  1  for  test  titles. 


Factor  Descriptions 

The  factor  descriptions  for  the  Study  2  analysis  arc  identical  to  those  for  Study  1,  with  the 
following  exceptions:  1)  the  Working  Memory  factor  is  now  defined  by  spatial  and  verbal  tests,  in 
addition  to  quantitative  tests;  2)  the  Processing  Speed  factor  is  a  little  weaker  because  the  Meaning 
Identity  test  was  no  longer  in  the  battery;  and  3)  the  Numerical  Facility/Perceptual  Speed  factor 
contains  less  numerical  variance  because  the  Number-Fact  Retrieval  test  had  been  dropped. 
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APPENDIX  C 

VALIDITIES  OF  ASVAB  AND  LAMP  TESTS  FOR  STUDY  1  AND  STUDY  2  ACCURACY  AND 

LATENCY  CRITERIA 


Table  Cl.  Validities  of  ASVAB  and  LAMP  tests  for  Study  1  Accuracy  Criteria 

BLOCK  AVERAGES 

TEST  Avl  Av2  AVG 


ASVAB 


GS 

.06 

.15 

.13 

AR 

.38 

.33 

.40 

WK 

.13 

.22 

.20 

PC 

.00 

.20 

.16 

NO 

.12 

.03 

.08 

CS 

,12 

.06 

.09 

AS 

.05 

.12 

.10 

MK 

.34 

.36 

.40 

MC 

.20 

.31 

.29 

El 

.16 

.28 

.26 

LAMP 

CRA 

.25 

.24 

.27 

NIA 

.26 

.25 

.29 

PIA 

.28 

.21 

.27 

DSA 

.31 

.28 

.32 

NFA 

.18 

.16 

.19 

NRA 

.41 

.43 

.47 

MIA 

.33 

.31 

.35 

ABA 

.40 

.40 

.44 

NAA 

.47 

.39 

.48 

CRL 

.06 

.02 

.05 

PIL 

.11 

.08 

.11 

NFL 

.03 

.09 

.07 

MIL 

.10 

.02 

.07 

ABL 

.28 

.34 

NIL 

.09 

.06 

.09 

Notes.  Block  codes  are  defined  in  Figure  2.  Test  abbreviations  arc  defined  in  Figure  1. 
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Table  C2,  Validities  of  ASVAB  and  LAMP  tests  for  Study  1  Latency  Criteria 

BLOCK  AVERAGES 


TEST 

Avl 

Av2 

AVG 

ASVAB 

GS 

-.04 

.04 

-.01 

AR 

-.13 

.02 

-.06 

WK 

-.06 

.01 

-.03 

PC 

-.04 

-.01 

-.03 

NO 

-.10 

-.12 

-.13 

CS 

-.10 

-.08 

-.10 

AS 

-.03 

.05 

.01 

MK 

-.10 

.00 

-.06 

MC 

-.01 

.01 

.03 

El 

.00 

.07 

.05 

LAMP 

CRA 

.05 

-.01 

.05 

NIA 

.03 

.06 

.07 

PIA 

.04 

.10 

.08 

L)SA 

-.12 

-.04 

-.07 

NFA 

,(X) 

.07 

.04 

NR  A 

-.14 

-.01 

-.07 

MIA 

.05 

.17 

.14 

ABA 

-.05 

.12 

.03 

NAA 

-.17 

-.04 

-.11 

CRL. 

.16 

.19 

.14 

PIL 

.33 

.29 

.32 

NFL 

.28 

.34 

.31 

MIL 

.19 

.25 

.21 

ABL 

.23 

.40 

.35 

NIL 

.35 

.34 

.36 

Notes.  Block  codes  are  defined  in  Figure  2.  Test  abbreviations  are  defined  in  Figure  1. 
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Table  C3. 

Validities  of  ASVAB  and  LAMP  Tests  for 

Study  2  Accuracy  Criteria 

TEST 

Avl 

2-1 

BLOCK  AVERAGES 

Av3  Av4 

AVC. 

ASVAB 

GS 

.13 

.21 

.20 

.22 

.21 

AR 

.33 

.35 

.33 

.26 

.34 

WK 

.11 

.15 

.19 

.16 

.17 

PC 

.07 

.11 

.11 

.13 

.12 

NO 

.14 

.14 

.14 

.12 

.15 

CS 

.13 

.10 

.07 

.08 

.11 

AS 

-.03 

.13 

.13 

.08 

.07 

MK 

.40 

.33 

.32 

.30 

.38 

MC 

.21 

.27 

31 

.26 

.29 

El 

.08 

.20 

.20 

.19 

.18 

LAMP 

GKA 

.14 

.15 

.14 

.16 

.17 

FRA 

.34 

.26 

.28 

.34 

.36 

MCA 

37 

.29 

.34 

.39 

.40 

NIA 

.20 

.25 

.22 

.20 

.23 

PIA 

.25 

.18 

.20 

.19 

.24 

CRA 

.30 

.17 

.20 

.26 

.28 

ABA 

.39 

.36 

.38 

.41 

.44 

WAA 

.35 

.31 

.33 

.36 

39 

NAA 

.44 

.36 

.36 

.43 

.46 

WKL 

-.10 

-.16 

-.16 

-.15 

-.15 

NIL 

-.11 

-.07 

-.07 

-.11 

-.11 

PIL 

-.09 

-.04 

-.05 

-.08 

-.08 

ABL 

.24 

.21 

.24 

.33 

.30 

CRL 

-.12 

-.17 

-.17 

-.18 

-.17 

Notes.  Block  codes  defined  in  Figure  2.  Test  abbreviations  defined  in  Figure  1.  The  values  under  2-1 
arc  for  a  single  block  of  simple  gates  with  negations. 
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Tabic  C4.  Validities  of  ASVAB  and  LAMP  Tests  for  Study  2  Latency  Criteria 

BLOCK  AVERAGES 


TEST 

Avl 

2-1 

Av3 

Av4 

AVG 

ASVAB 

GS 

-.19 

-.15 

-.18 

-.07 

-.19 

AR 

-.23 

-.17 

-.14 

-.09 

-.18 

WK 

-.09 

-.01 

-.04 

-.02 

-.06 

PC 

-.06 

-.06 

-.09 

-.01 

-.07 

NO 

-.14 

-.09 

-.10 

-.13 

-.12 

CS 

-.08 

-.06 

-.13 

-.14 

-.11 

AS 

-.02 

-.09 

-.01 

-.00 

-.02 

MK 

-.22 

-.14 

-.14 

-.06 

-.17 

MC 

-.14 

-.11 

-.05 

.01 

-.08 

El 

-.09 

-.04 

-.05 

.05 

-.05 

LAMP 

GKA 

-.22 

-.17 

-.19 

-.10 

-.21 

FRA 

-.22 

-.13 

-.13 

.01 

-.15 

MCA 

-.23 

-.14 

-.11 

.04 

-.13 

NIA 

-.09 

.02 

.04 

.01 

-.02 

PIA 

-.02 

.05 

.08 

.03 

.04 

CRA 

-.03 

.02 

.06 

.18 

.07 

ABA 

-.17 

-.07 

-.03 

.07 

-.07 

WAA 

-.27 

-.09 

-.11 

.07 

-.14 

NAA 

-.27 

-.16 

-.13 

.06 

-.14 

GKL 

.26 

.24 

.25 

.18 

.27 

NIL 

.38 

.25 

.35 

.22 

.37 

PIL 

.37 

.28 

.36 

.25 

.38 

ABL 

.10 

.22 

.26 

.41 

.29 

CRL 

.33 

.26 

.23 

.15 

.28 

Notes.  Block  codes  defined  in  Figure  2.  Test  abbreviations  defined  in  Figure  1.  The  values  under  2-1 
are  for  a  single  block  of  simple  gates  with  negations. 
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